Quentin Ambard

#DataEngineering 𝗣𝗿𝗼 𝗧𝗶𝗽𝘀 𝗳𝗿𝗼𝗺 𝗙𝗮𝗯𝗿𝗶𝗰𝗲 𝗗𝗲𝘀𝗲𝘆𝗻, 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 𝗖𝗵𝗮𝗺𝗽𝗶𝗼𝗻 Join our colleague, Fabrice Deseyn, for a deep dive into the world of Databricks. In a recent interview with Youssef Mrini on the #NextGenLakehouse YouTube channel, Fabrice shares his insights and journey of becoming a Databricks champion. Beyond his champion status, Fabrice provides expert guidance on navigating Unity Catalog migrations. Discover practical tips and strategies to streamline the process and ensure a successful transition. Watch the video now 👉 https://lnkd.in/e2zFw53p #Databricks

11

1 Comment

This is for REALLY advanced Airflow users The presentation is impressive👇 My friend Philippe Gagnon did a talk about the Airflow Scheduler. This kind of insight is impossible to find anywhere else. Here are the key takeaways 🔑: ➡️ The main scheduler loop is where everything happens (initialization of loop timers, which are intervals of time performing maintenance operations on the objects the scheduler works with - task instances, datasets, etc / executor and task instances heartbeats). ➡️ max_tis_per_query has a large impact on scheduler performance (default 16). Large values will cause each iteration of the scheduling loop to take longer (slowing the queuing process) but they will perform more work. ➡️ Task instances are selected among running DAG runs and ordered by priority weight, which is a global value that spans all currently running DAG runs. ➡️ Task scheduling priority is decoupled from DAG run scheduling priority (because of priority weight). Priority weight is calculated according to a weight rule. ➡️ Default weight rule is "downstream." The task's effective weight is the aggregate sum of all downstream descendants. Therefore, the structure of a DAG impacts task scheduling priority. For example, DAG A is a very deep DAG (one after the other) that coexists with DAG B (only three tasks in parallel at the top), which is a very shallow DAG. If DAG A is scheduled very aggressively, it's possible that task instances in DAG B will never be scheduled depending on max_tis_per_query. That can make you think DAG B is starved whereas it's not. and more! If you're interested in understanding the intrinsic details of the Airflow scheduler, go to the official Apache Airflow YouTube channel. Enjoy ❤️ #airflow #apacheairflow #dataengineering #dataengineer

89

1 Comment

Here's a secret Snowflake tip 🤫: with the ability to host Notebooks on the new container runtime, we can now install arbitrary packages using !pip. 🔥 Yes, we can also install packages not available on the Anaconda Snowflake repo (with all the risks that come with it 😅). For instance, now you can install the Cinemagoer package and search the IMDb for all the movies containing the word "matrix" in the title 👇

92

3 Comments

Remember back in 2012, when PostgreSQL introduced JSON support ? https://lnkd.in/e_Qb8rpc Now in PostgreSQL 17 we have JSON_TABLE and new JSON functions !

3

Data Engineers ! This one's for you: 🧠 The Economics of Generative AI Stephanie Kirmer 🧠 Connecting ML Models in Sagemaker to Snowflake and dbt Hugo Lu 🧠 Introduction to Interpretable Clustering NAKUL UPADHYAY 🧠 Snowflake automatic clustering cost estimation Tim Sander 🧠 Death to Tech Debt? Ari Joury, PhD Check it here; https://lnkd.in/ebdFKNZ5 #dataengineering #dataroundup #startup

5

📢 New Video Alert: Keynote by Julien Le Dem on Apache Iceberg and the Deconstructed Database 🎥 Watch Julien's keynote from the #ChillDataSummit in #SanFrancisco. Julien Le Dem is one of the key innovators in the open-source community and a driving force behind projects like Apache Parquet. In this talk, Julien explores how Apache Iceberg is revolutionizing data lakes, breaking down the concept of the deconstructed database and its impact on large-scale data management. If you're passionate about data architecture or curious about the future of data processing, don’t miss Julien’s insights on how Iceberg is pushing the boundaries of what's possible in data lakes. 📽️ Watch the full video here: https://lnkd.in/d9eg2exq #ApacheIceberg #OpenSource #DataEngineering #DataLakes #BigData

26

1 Comment

Build an effective recommender system seamlessly within your Snowflake environment using RelationalAI’s Snowflake Native App! Curious to see how it works? Check out this solution architecture on Snowflake Solution Center!

45

Let's transform raw page view data from Apache Kafka into aggregated insights directly loaded into Databricks Delta tables. 🧱 We explore real-time data aggregation, demonstrating how to downsample page view data across a 10-second time window using just a few SQL queries. By processing data before it reaches Databricks, you can dramatically reduce storage and compute costs while minimizing latency. It's all here 👇 https://lnkd.in/g7QwbSUD

3

🔎 Looking for a better Kafka? Search no more and join my session at Big Data Conference Europe on November 22. I’ll be presenting Apache Pulsar at 11:15 CEST online. Register using the link below in the comments. #ApachePulsar #ApacheKafka #DataStreaming

21

1 Comment

I presented about Apache XTable's architecture last week at LinkedIn's Big Data Meetup. For context, XTable is an omni-directional translation layer on top of open table formats such as Apache Hudi, #ApacheIceberg & Delta Lake. It is NOT ❌ a new table format! Essentially what we are doing is this: SOURCE ---> (read metadata) ---> XTable's Model ---> write into TARGET We read the metadata from the SOURCE table format, put it as a unified representation & write the metadata in the TARGET format. * Note that we are only touching metadata, not the actual data files (such as #Parquet) with XTable. Let's breakdown its architecture. XTable’s architecture consists of three key components: 1. Conversion Source: ✅ These are table format specific modules responsible for reading metadata from the source ✅ They extract information like schema, transactions, partitions & translate it into XTable’s unified internal representation 2. Conversion Logic: ✅ This is the central processing unit of XTable ✅ It orchestrates the entire translation process, including initializing of all components, managing sources and targets, among other critical things 3. Conversion Target: ✅ These mirror the source readers ✅ They take the internal representation of the metadata & maps it to the target format’s metadata structure Read the paper (link in comments) for details. #dataengineering #softwareengineering

199

5 Comments

The first milestone to integrate Large Language Models with Apache Solr has been set! This introduces in the upcoming Solr 9.8 release the ability to use external text to vector inference models, directly in your textual query parser! This means that you will be able to run semantic search via text (and Solr will encode to vector and run vector search behind the scenes). This is a first step and a huge enabler, which I've been happy to contribute! I hope that this will open the doors to many more experiments, applications, bug fixes and new contributors! This wouldn't have been possible without the support of Cedric Ulmer and the help of fellow committers David Smiley, Christine Poerschke, Eric Pugh and Christos Malliaridis! What's coming next? In early 2025 the indexing side, to give the possibility of encoding vectors transparently, while indexing. And then? Retrieval Augmented Generation, from natural language, to structured query, cross encoders rerankers and more! Stay tuned!

111

1 Comment

Shift left from Looker to dbt™: Leverage Euno to centralize your metrics. Automatically generate open-source dbt-semantic (MetricFlow) code from selected Looker/LookML measures, centralizing all your metrics in dbt's metrics mart. See Euno in action: https://lnkd.in/dzvT9N95

12

Attention Snowflake users ! Wouldn't it be great if every time anything anomalous happened in your infrastructure it got flagged? Well in Orchestra that's pretty easy to set-up. Check it out below! https://lnkd.in/eP8bj6MU #dataengineering #anomalydetection #snowflake

7

Why wait for slow queries when you can speed things up? On the Apache Arrow blog, Ian Cook, David Li, and Matt Topol, break down how Arrow's format accelerates query result transfers by eliminating inefficiencies. Key takeaways: ✅ Columnar format reduces costly transpositions ✅ Zero-copy architecture minimizes overhead ✅ Universal language support ensures seamless integration across platforms Read the full post to learn how Arrow boosts query performance and discover real-world speedups from tools like Theseus, DuckDB, and Snowflake. 🔗 Read more: https://lnkd.in/eBbjz5uZ #DataAnalytics #ApacheArrow #BigData #OpenSource

18

Postgres has a highly extensible architecture, but it needs to be modified to support large-scale analytics in order to handle the AI workloads of the future. That was one of the key topics at hand as EDB Senior Director Franck Sidi sat down with EDB Technical Fellow Dr. Marc Linster to discuss the relationship between AI and analytics, as well as how to apply AI techniques to enhance or automate analytics. In order to support large-scale analytics, Postgres has to be extended in these four ways: 1. Decoupling compute and storage - Decoupling the compute engine with the storage engine to allow independent scalability. 2. Dynamic scalability - Horizontal and dynamic scalability for compute with a distributed query engine. The same query can be split and run on multiple computer nodes using a distributed query execution engine. 3. Columnar storage - Support for Columnar Storage format for the tables is essential for query execution speed to access only relevant columns. This also enables efficient data compression as you can apply compression algorithms per column. 4. Vectorization - A vectorization query engine makes it possible to leverage the CPU modern chipset. Vectorization means that you can work on multiple rows in 1 CPU instruction. Single Instruction, Multiple Data (SIMD concept) Want to dive deeper into using Postgres for analytics, search, and AI? Read the full blog here ➡️ https://bit.ly/3ZsbYYL #EDBPostgresAI #JustSolveItWithPostgres #AIAnalytics #LargeScaleAnalytics #PostgreSQL #VectorDatabase

34

1 Comment

🚀 KAWA Analytics is the easiest way to build enterprise-grade data apps on Snowflake From risk management to dynamic pricing, ESG to market abuse surveillance, KAWA AI Co-builder makes it effortless. 🎥 Watch our product manager Narjisse Bouri demonstrate how to build a market abuse monitoring system in no time.

5

1 Comment

SingleStore Announces Bi-Directional Integration with Apache Iceberg Learn more & get our take 👇 https://lnkd.in/gNTBCWAE "Our vision has always been to provide one single data store for all companies to be able to take advantage of speed, scale and simplicity. Our data platform is designed to unlock all types of enterprise data — including data that is frozen in data lakes." — Raj Verma, CEO at SingleStore "SingleStore’s data platform is indispensable for analyzing our data and delivering real-time insights to our customers. This latest update continues to push the boundaries of what we can accomplish by harnessing all of our enterprise data in real time." — Mauricio Aristizabal, Principal Data Architect and Founder at Giist Social "The integration with Iceberg, plus enhancing our platform’s scaling and processing capabilities will allow users to access and harness their data in real time, and build intelligent apps." — Nadeem Asghar, Head of Engineering at SingleStore #apache #product #SoTNews

1

🚀 𝗘𝘅𝗰𝗶𝘁𝗲𝗱 𝗳𝗼𝗿 𝘁𝗵𝗲 Databricks 𝗪𝗼𝗿𝗹𝗱 𝗧𝗼𝘂𝗿 – 𝗡𝗲𝘅𝘁 𝗪𝗲𝗲𝗸! 🌍     Catch us at 𝗕𝗼𝗼𝘁𝗵 𝗚𝟭𝟭! 🗓️ We can’t wait to engage in dynamic 𝗰𝗼𝗻𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗼𝗻𝘀 𝗼𝗻 𝗵𝗼𝘄 𝘁𝗼 𝗲𝗹𝗲𝘃𝗮𝘁𝗲 𝘆𝗼𝘂𝗿 𝗱𝗮𝘁𝗮 𝘀𝘁𝗿𝗮𝘁𝗲𝗴𝘆 𝗮𝗻𝗱 𝘂𝗻𝗹𝗲𝗮𝘀𝗵 𝘁𝗵𝗲 𝗳𝘂𝗹𝗹 𝗽𝗼𝘁𝗲𝗻𝘁𝗶𝗮𝗹 𝗼𝗳 𝘆𝗼𝘂𝗿 𝗱𝗮𝘁𝗮. Here are some inspiring topics that we would like to explore and share insights with you during our exchange at the booth:     🔍 Unlocking Enterprise Data for GenAI applications  📊 Assessing the maturity of your data landscape  💡 Developing GenAI use cases to maintain a competitive advantage in your industry  🛠️ Data governance and management best practices  📈 Understanding data traceability and lineage  📜 Crafting effective data management policies  ⚙️ Building infrastructure that maximizes business value    We can’t wait to 𝗰𝗼𝗻𝗻𝗲𝗰𝘁 𝗮𝗻𝗱 𝗵𝗮𝘃𝗲 𝘀𝗼𝗺𝗲 𝗶𝗻𝘀𝗽𝗶𝗿𝗶𝗻𝗴 𝗰𝗼𝗻𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗼𝗻𝘀 at our booth G11! ✨     #Databricks #DatabricksWorldTour #Data #DataFoundation  

18