AI Engineer | Python LLM Systems | Agent Harnesses | Multi-Agent Orchestration

Currently I have an ongoing project for a mid size German e-commerce company. I am automating their manual repetitive input, which has been established for 10+ years and grown to a painful level - It is about CSV/PDF data extraction using LLMs(claude) and upload to JTL(e-comerce shop) via API.

Recently I helped a client of mine with automations on sales calls, by providing him with quick access to information. I extended his (basic) RAG agent by adding more complex ChatGPT workflow, Pinecone, scraping and connected it to a web u. Here is a video I made for him, explaining how to connect this agent:  https://youtu.be/ghB4_SrqHT4

Client was struggling with high costs and slow transcription turnaround. I completely transformed the workflow by optimizing the speaker diarization process and WhisperX integration. The results were immediate - processing time was cut in half while maintaining excellent accuracy. Users now get their transcriptions much faster and client is saving significantly on operational costs.

The primary objective of our project was to design, build, and optimize data pipelines that could effectively handle large datasets, transform them, load them into S3 buckets, and distribute them across multiple tables in BB. Our goal was to create scalable and fault-tolerant data processing solutions that could meet the needs of various projects and departments within the company.

As a Data Engineer, I collaborated closely with data scientists and business stakeholders to understand their data needs and designed data pipelines that met those needs. I built, tested, and maintained the pipelines, ensuring that they were optimized for performance and could handle large datasets. Furthermore, I monitored the performance of the pipelines, identifying bottlenecks and issues, and implementing solutions when necessary.

To achieve our goal, I leveraged AWS Cloud services such as S3, EMR, and EC2 to build scalable and fault-tolerant data processing solutions. These solutions helped ensure that our data pipelines were capable of handling large datasets and distributing them across multiple SQL tables efficiently. Additionally, the use of AWS Cloud services enabled us to make the pipelines scalable and fault-tolerant, thereby minimizing downtime and increasing overall performance.

Thanks to our combined efforts, the data pipelines were able to successfully handle large datasets and distribute them across multiple SQL tables. Our optimized and scalable data processing solutions helped to minimize downtime and increase overall performance, resulting in a successful project that met the needs of various projects and departments within the company.

I developed a telegram bot for @tbilisieventss. This bot helps Users to get all the required information as well as helps admins to provide content.

Technologies:
- python
- python-telegram-bot
- sql
- docker

The primary objective of our project was to capture market anomalies in commodities futures markets, build a trading system around it, and optimize it for hit rate to improve the quality of trades. To achieve this goal, we leveraged historical and real-time data from multiple sources and implemented advanced machine-learning algorithms to train our predictive models.

As a key contributor to the project, I designed and implemented pipelines to collect, preprocess, and load historical data and real-time feeds from multiple exchanges into a NoSQL database, specifically MongoDB for time-series data. I also implemented web sockets to capture real-time events from different APIs, exchanges, and news feeds buffered them for the algorithm and channeled them through a pipeline.

Furthermore, I engineered predictive features to train a machine learning algorithm, specifically Random Forest and optimized it for hit rate to improve the quality of trades. I developed Grid Search algorithms for parameter optimization and created Cython modules to improve computation efficiency and resolve performance bottlenecks. Additionally, I integrated and tested various elements into an existing framework and ecosystem through a CI/CD pipeline to ensure the system was robust and reliable.

Thanks to these efforts, the trading system built using these technologies and methodologies was a tremendous success. By leveraging historical and real-time data from multiple sources and using advanced machine learning algorithms, the system was able to optimize for hit rate and improve the quality of trades. The use of Cython modules for computation efficiency and the CI/CD pipeline for testing and integration ensured that the system was blazingly fast, robust, and reliable.

Overall, the project provided valuable insights into market anomalies, allowing us to capitalize on opportunities that would have otherwise gone unnoticed. Through our combined efforts, we were able to deliver exceptional results, and the project served as a testament to the power of advanced machine learning algorithms and well-architected data infrastructure.

As a Data Engineer at our startup, my primary responsibility was to create a scalable trading infrastructure that could handle large amounts of historical and live data feeds from various exchanges. I accomplished this by collecting, preprocessing, and loading the data into NoSQL databases such as MongoDB, InfluxDB, and Redis. Additionally, I developed our API using fastAPI, a high-performance web framework for building APIs in Python, which allowed us to integrate our system with external applications seamlessly.

As startups often require a "jack of all trades," I also designed multiple trading systems using the LEAN engine from QuantConnect. This engine enabled us to easily create and test algorithms and deploy them into production environments without any mental overhead. To ensure maximum efficiency, we deployed these systems in Docker containers on our servers and orchestrated them using simple Docker compose and bash scripting.

Although our startup failed for reasons beyond my control, I gained invaluable insights into software development during this period. I learned to appreciate the importance of CI/CD tools for team coordination and transparent deployment. I also discovered the usefulness of well-known Design Patterns, which helped me avoid reinventing the wheel and improve the readability and stability of my code. Additionally, I learned the hard way about the importance of "composition over inheritance," which I had used in the wrong place at the wrong time, resulting in multiple rewrites and wasted time.

Overall, my experience at the startup taught me many valuable lessons, which have made me a more skilled and experienced data engineer.