Profilbild von Dmitry Vasilets Big Data and DevOps Engineer aus Heidelberg

Dmitry Vasilets

teilweise verfügbar

Letztes Update: 01.10.2021

Big Data and DevOps Engineer

Abschluss: nicht angegeben
Stunden-/Tagessatz: anzeigen
Sprachkenntnisse: deutsch (gut) | englisch (verhandlungssicher)


Contributor to many opensource projects: pulp, apache-spark, ovirt, kubernetes, kubernetes/kops, activemerchant, spree, ranger, logstash, theforeman, fedora, katello, katello/forklift, vagrant-libvirt, firewalld, glusterfs-container. Big data infrastructure: hadoop, mapreduce jobs, hive, sqoop, oozie,ambari. Big data analytics: SVM, logical regression, k-mean, pagerank, cart. Big data visualization: dc.js(d3.js + crossfilter or rChart ) + R(charts, statictic), graph-databases. Cluster computing: yarn(hadoop), apache spark, mesos. Microservices architecture tools: mesos, kubernetes, openshift. Stream computing: apache storm, stream-sql. Data science: prediction, statistics, process mining, data mining. web servers: apache, nginx – configuration and patching for add new features. Databases:mysql, postgresql, ha, horizontal and vertical scalability. NoSQLDB: hbase, couchdb, mongodb, cassandra, elasticsearch, Neo4J Queue systems: kafka apache, amqp(rabbitmq,qpid), hornetq Virtualization (libvirt, xen, kvm,vmware,vagrant, OpenStack) Automatic deploy using chef,ansible and puppet Spring: boot, cloud. VoIP (asterisk, openser) monitoring by snmp ,opennms, nagios, zabbix, cacti, collectd,rrd,prometheus, grafana ELK contributing and customizing. standart network services (ftp,dns,samba,dhcp,proxy,firewalls,nfs) IDS systems (snort, suricata, cisco) Expert in svn and git Scripting (ruby, lua, bash, awk, python) Statistics: R, sas, python Vissualisation: js(d3.js, crossfilter, dc.js, nvd3), octave, R(rchart, graphics), blender (by python api) Experience with Scala, C, Go, erlang, java


07.2018-02.2019 younicos(Aggreko)
Self managed powergrid system.
Establish release process. Create self managed system based on openshift(kubernetes), glusterfs, openstack swift, activemq, ansible, ospf, karaf, prometheus,elasticsearch, kibana.
Create ansible module for moxa network devices. VPN connections based on moxa edr and cisco ASA devices.
Create timeserver based on gps receiver.

08.2018-2.2019 CGM AG
Integrate IDS to existed infrastructure based on suricata, puppet, foreman, netflow, elk stack . Upgrade to IPS. Design SIEM (security information and event management).

07.2017-08.2018 EOS
Create automatized distributed infrastructure for datascientists from different countries.Create system for full recovery from scratch system for big data analytics teams(multitenancy, authentiction, authorisation, audit, kerberos, encryption by default). Patching ambari and hortonworks services(nifi, metron, ranger).Configure Oracle Big Data Sql for working together with Hortonworks hadoop and Oracle Exadata. Preparation for GDPR (DSGVO): anonymisation, taging and masking sensitive data, audit, autodeletion. Integration to company infrastructure(ActiveDirectory binded with Freeipa pki, kerberos and ldap, network services like dns, dhcp, vlans). Integration with infoblox services. Automatic installation and configuration by ansible,puppet and foreman(redhat satellite). Cloud solution based on openstack. Manageiq(redhat cloudforms) for chargeback and integrate different cloud, containers and infrastructure providers(openstack, openshift, vmware, ovirt).

08.2017-present CTO,Founder
Create self serviced infrastructure. Create authentication, authorization and accounting system. Manager dell hardware by idrac thought api.
AI + DevOps services. Robot which replace operators and network administrators.
Distributed system for manage servers, network device and cloud. Based on openshift, hadoop, nifi, theforeman, freeipa, manageiq and openstack.

10.2015-present Mentor on course “Data Engineer”,
train people to use big data tools for real cases.
Create cases for spark, storm, graphx + neo4j.
Graduated persons successfully working in big data fields.
PAAS based on openshift with distributed storage on glusterfs(managed by heketi) with share cpu,gpu and network resources.

05.2017-06.2017 Marketlogic. Automatic deploy openshift to aws,baremetal and openstack. Create pipelines for rolling update applications. Adopt RoleBasedAccessControl for company requirements. Integrate openshift cluster with cloudera cluster and define network rules and policies for that.

03.2017-04.2017 Elinvar. Automatic delpoy kubernetes infrastructure in aws with encrypted all data and networks. Kubernetes, ipsec, aws: ec2, ebs, vpc, kms, elb, weave, k8s/kops, kafka, tls.

04.2017,07.2018 MotionLogic. provided big data infrastructure audit. Created roadmap for improve security, stability and automatisation. Hortonworks data platform, freeipa(ldap, kerberos), oracle linux, numpy, containers and virtual network.
Create roadmap for new product based on neo4j and spark.

06.2016-03.2017 Strato AG. Software cloud engineer.
Design,implement and integrate to cloud infrastructure based on freeipa(ldap, kerberos, pki, tls) , theforeman+katello projects , openstack.
Create ci pipeline for manage cloud, integration tests for salt formulas, puppet modules, ansible playbooks. build system for rpm packages and create infrastructure for rpm repositories and docker images(crane + pulp + foreman). Create high availability infrastructure with automatical deployment openstack.
Distributed storage: glusterfs and Ceph. Integrate central authentication, authorization and certificates infrastructure in HA mode (FreeIPA: dogtag, ldap, kerberos, selinux, vault, dnssec). Security scaner OpenScap for fit standart pcidss.
Provide internal trainings for collegues.
Write patches for foreman, inspec, flask, hammer, katello, pulp, puppet, openstack.

08.2015-03.2016 ExacTag – Duisburg (contractor)
Cloudera Hadoop, Kafka, Spark(graphX, streaming)
Create realtime application for advertisement metrics
Monitoring(zabbix, jmx) and tune spark application
Patching spark streaming and RDD for simplify data aggregation.
Cassandra as raw data storage.

05.2015-08.2015 IngDiBa Bank – Frankfurt am Main (contractor)
Integrate together: Hortonworks Hadoop, Ranger,RedHat Satellite, FreeIPA, Ambari, Spark,HDFS, Hive, Spark, Yarn, Hbase, Kerberos
Rstudio, R , local CRAN.
Docker and docker-registry for local distribution docker images.
Authorisation and authentication in hadoop with kerberos and ldap.
Create big data platform for data scientists.
Fraud protection.
Final result is

01.2015-05.2015 Fujitsu – Munich (contractor)
Development FUJITSU Software ServerView Cloud Load Control
OpenStack(neutron, nova, cinder, heat)
OpenShift,ProjectAtomic(multinodes cluster), kubernetes (multinodes cluster inside Atomic), docker
Web Management tool based on angularjs + d3.js + java + gradle,jax-rs

05.2013-04.2015 Citozin – Berlin (Internet of Things)
founder, backend engineer
collect and analyze data (fluentd, cassandra, postgresql, hadoop, R , sqoop, d3.js)
openstack + hadoop(hortonworks)+centos
Multiple point for write to cassandra.
Kafka as message broker.
Cars error prediction based on collected data and logit model.
DWH design and implement ETL process.
Hardware device creation – bluetooth low energy, OBD2 protocols(mainly CAN)
create product(docs, ads, manage team)

02.2014-12.2014 Nokia-Here – Frankfurt am Main (contractor)
ruby, python, rails , postgresql(json, hstore), aws ec2, ebs, glacier, d3.js, crossfilter.js, dc.js
data visualization by graphics and heatmaps, agregations by geographic and roles createria
log analyze by splunk and logstash
elasticsearch cluster(>10TB): rebalancing, scaleup, failover
create custom logstash filters.
Hadoop + elasticsearch for aggregation.
Go for write REST api
R + shiny + splunk api for draw beauty and fast graphics.
Cloudformation templates for autodeployed and scaled elasticsearch cluster
develop on java and jruby – improve and speedup logstash for s3(opensourced on github)
Write puppet modules.

01.2013 – 08.2013 DevOp in Zimory GmbH
  1.implement distribution product by rpm(maven for build rpms)
  2.automatize installation process(puppet + foreman + rpm)
  3.implement central log system(puppet + logstash + rsyslog)
  4.testing and bug fixing(vagrant + vagrant-libvirt + python + ruby + bash)
  5.ldap modification
  6.Message system on HornetQ

04.2012-11.2012 Hitfox GmbH
chef's cookbooks, spree extentions, tests, support infrastructure
fast(<60ms) search product on eshop
maintainane and improve  hadoop cluster.
Map reduce jobs.
Realtime advertisement
Improve speed of services.

10.2011-02.2012 PAYANGO GmbH
Prepaid card service.
Card printer api integration. Security improvements.
chef integration,write all recipes and migrate part of system from scalarium(opsworks, chef)
vagrant integration for autodeploy from chef-server
failover system for amqp(based on rabbitmq)

10.2007 – 05.2014 Pronix

data engineer,devops engineer  and manager

- build system for draw geodata by d3.js
maintainer of project vagrant-libvirt and many spree extentions
contributor to projects: foreman, activemerchant, ovirt
ui based on angularjs and create deploy recipes, ha cluster(
ember.js based application(js + html5)
rails and backbone code for
postgresql cluster based on hot standby feature and heartbeat.  - spree based e-shop with geo-settings for dealer network.
create autocomplete documents service
online store(spree and spree plugins
) filehosting service(hadoop(hbase + map/reduce tasks, realtime configuration cluster based on traffic storm apache) + rails + memcached)
jruby client app + hadoop
openstack as private cloud.
jruby for facebook automatization
jruby for deploy rails application on google app engine
jruby for amazon orders serve
eshop + remote soap datastorage (1c)
realty information service (rails + couchdb + memcached)
increase performance for>6_000_000 requests per hour)
nginx and varnish for cache web pages and speed up applications.

system administrator
Patching mysql, apache, nginx.

freelancer,system administrator and ruby developer
create and manage LAN for campus with 3 buildings.
Create webui for manage it.
Integrate IDS for increase security.

Zeitliche und räumliche Verfügbarkeit

Germany mainly