Overview

Data Engineer Jobs in Sri Lanka at CrediLinq.Ai

Title: Data Engineer

Company: CrediLinq.Ai

Location: Sri Lanka

We are seeking a skilled Data Engineer to manage, optimize, and scale our modern data ecosystem. Our core ingestion pipelines and data lakehouse architecture are already built; your mission will be to maintain these systems, optimize performance, and ensure high availability.

We are specifically looking for a Data Engineer who understands how to manage event-driven, on-demand scalable pipelines. You will help manage our cloud infrastructure, ensuring our serverless Prefect platform on AWS ECS runs smoothly and scales efficiently based on workload demands. While our data pipelines are orchestrated via Prefect, we welcome strong Airflow engineers who are ready to adapt. Additionally, as we continue to innovate, exposure to or a strong interest in AI Engineering and Agentic AI workflows will give you a significant edge.

Key Responsibilities

Data Infrastructure & Pipeline Maintenance

  • Orchestration & Scale: Maintain, optimize, and continuously improve our existing event-driven, on-demand scalable pipelines within the Prefect ecosystem. Ensure reliability and scale handling data ingested from Amazon SP-API, eBay, and Plaid.
  • Data Lakehouse Management: Maintain our Medallion Architecture using DuckDB/DuckLake, ensuring historical data and real-time snapshots are accurately centralized.
  • Cost & Performance Efficiency: Monitor and tune our per-minute serverless execution model to ensure zero idle-compute costs while handling high-throughput, on-demand data loads.

Cloud, DevOps & Platform Ownership

  • AWS Platform Management: Support the deployment and management of Prefect infrastructure and agents across AWS services, with a focus on AWS ECS.
  • CI/CD Pipelines: Maintain and streamline deployments using Bitbucket Pipelines.

AI Support & Integration (Growth Area)

  • Semantic Layer Support: Maintain the semantic data layer that feeds into our downstream AI workflows.
  • AI Collaboration: Assist in supporting production-grade infrastructure (GCP Vertex AI / AWS ECS) that powers our NL-to-SQL capabilities and batch inference pipelines for transaction categorization.

Technical Skills & Requirements

Qualifications & Experience:

  • 2+ years experience in data engineering.
  • Core Languages: Strong proficiency in Python and SQL is essential for maintaining our processing scripts and optimizing database queries.
  • Pipeline Architecture: Proven experience working with event-driven architectures and building/maintaining on-demand, scalable data pipelines.
  • AWS & Infrastructure: Good experience with AWS cloud infrastructure—particularly managing and troubleshooting containers via AWS ECS to support our orchestration platform, alongside hands-on experience with Amazon S3 and Lambda functions.
  • Workflow Orchestration: Production experience with Prefect is a plus, but strong experience with Apache Airflow (and a willingness to learn Prefect) is completely welcome.
  • Data Warehousing: Experience with DuckDB/DuckLake is ideal, but a solid background with any modern columnar data warehouse/lakehouse (e.g., Snowflake, BigQuery, Redshift, ClickHouse) is good enough to maintain our current stack.
  • API Integration: Experience handling complex financial/e-commerce APIs (such as Plaid or Amazon SP-API).
  • DevOps: Experience managing CI/CD deployments via Bitbucket Pipelines or similar Git-based tools.

Nice-to-Have (The "Plus" Factors):

  • AI / Agentic Engineering: Knowledge or hands-on experience with Agentic AI systems, Large Language Models (LLMs), or NL-to-SQL frameworks is a definite plus.
  • Experience supporting machine learning infrastructure via Amazon SageMaker or GCP Vertex AI (for batch inference and time-series forecasting).

Apply

Upload your CV/resume or any other relevant file. Max. file size: 800 MB.