TDM S3 Ep4 - Ben Rogojan - Data Engineering Secrets: From Facebook to Freelance

And just like that, we're back with a candid discussion on the state of data engineering in 2025! Join us as we sit down with Ben Rogojan, aka the Seattle Data Guy, for a sharp analysis of the consulting landscape. With over a decade of experience in data infrastructure and analytics, Ben offers a thoughtful look at:

• The evolution of data engineering roles and skills
• Challenges faced by consultants in today's market
• Navigating the transition from full-time to consulting work
• Essential tools and technologies shaping the field

Don't miss this insider's perspective on cloud migration, AI integration, and the future of business intelligence. Whether you're a seasoned pro or considering a career shift, Ben's insights will help you stay ahead of the curve in data engineering.

Tune in for expert advice on building your personal brand, attracting clients, and thriving in the ever-changing world of data. This episode is packed with actionable tips and industry trends you won't want to miss!

#dataengineerinterview #dataengineer #analyticsinsight #dataanalyst #datamining

CHAPTERS:
00:00 - Intro
03:32 - The Data Fix
09:16 - Getting Ben Rogojan on the Show
12:09 - What's it Like to Work at Facebook
18:52 - Understanding Data Engineering
24:30 - Semi-Technical Expectations in Tech
27:30 - Transitioning from Technical to People Problems
29:37 - Your Role in the Data Lifecycle
30:31 - Identifying Your Clients
31:47 - Building Popularity in the Data Field
34:13 - Rise of Data Engineering Popularity
36:17 - Importance of Community in Data
37:53 - Growing Demand for Data Services
41:40 - Is Now a Good Time for Consultancy?
49:20 - Post-Show Discussion
49:57 - Qlik Acquires Upsolver
52:01 - DBT Labs Acquires SDF Labs
55:18 - Final Thoughts and Last Words
57:58 - Teaser for Next Episode

Transcript

[Music] your fix of the best guests from the world of data and analytics this is the data mix with Brian buen and George pon [Music] Brian George we're back high five oh oh gosh I'm might practiceing this way though it's always the opposite right yeah yeah nice one man good to see you how are you ah pretty good thank you I was up early this morning um it was a half past 4 start to go from my cold wet home in Scotland down to London the Big Smoke as we call it yeah I I think we should have like a like a we's Wally guess weGeorgia type of segment as well because like you're never in the same place twice mate but it's always always a pleasure to have you here and no 4 a.m. Journeys for me but I do have the plumber downstairs fixing the kitchen sink so um it's it never stops in the world of data and everything else right okay so if we hear some Plumbing noises going on we'll know what it is we'll just ignore them and move on um we have something much more exciting than uh office spaces and plumbing Today l right oh yes we do uh this is one I'vebeen looking forward to for a while I mean look I know I know we do say that but I think um this one's like kind of cool we're really excited that um that it's happening um obviously more to come uh we've got Ben Rogan here SE Seattle data guy and if you don't know who he is you're going to find out very very shortly we're going to bring him on but we have a few things just to um just to take on first George right so has become customary um we love your support guys thank you so much foreverything that you do for the show uh attending and you know giving us questions and putting stuff in the chat downloading us please rate our show okay for us to be able to bring um all these guests to you we work hard at it and frankly I think we deserve five star reviews right let's just let's just not hide behind that barrier anymore just give us the five star I agree I agree and then we we don't have to hassle you anymore for it in person I mean we'll still do it on the show but we won't doit in person anymore so um zap the Spotify and the Apple they're the ones that make the big difference for us or sub Us on YouTube as well if you've enjoyed rewatching the show back we appreciate you um big plug over um this is the segment of the show that's coming up that we call the data fix George right and let's get this let's get us rocking and rolling here are you ready for us I sure am all right let's go for it jingle we're heading into the data fix here we go so um we always give a little bit ofanalysis on what's happening in Tech and this is George's segment and George um you've got a good one this week right I'm looking forward to hearing it yeah so I want to talk this week about how uh politicized technology is coming um so I was uh listening to um some of my favorite um technology podcasts uh it wasn't this one um it was one of the other ones I like to listen to um and uh there's a couple of announcements um over the over the past couple of weeks um around diversity equity and inclusionprograms uh within big tech companies so these are companies like Amazon like meta um who are either cancelling them um or um at the very least reviewing them uh because they have got uh to walk um and two and two politicized as as they say um which I think there's some deep irony there I'm going to go back to um possibly before a lot of our listeners and viewers were born back to 2020 do you remember that far back um Brian oh pandemic e somewhere around yes well exactly so so back then was um chap called Brian Armstrong Brian Armstrongum was he may still be uh the CEO of coinbase um and back then he wrote this um uh what became fairly famous blog post um Around Mission focused so that was basically he felt back then in 2020 technology was getting too politicized and therefore he said okay we're going to focus on what we do best technology and we're going to ignore politics well fast forward a couple of years um um Mark Zuckerberg appeared on Joe Ro um Rogan's um show in 2022 talking about um how he was trying to walk a narrow path he was trying tobe the Arbiter of the truth um on the internet fast forward a couple more years are marks back um really lambasting the Biden Administration talking about how um they'd been hand they'd been Shackled they'd been shouted at down the phone by government officials telling them to um you know take things down um and the other thing that sort of uh jumps out at me is that um at the beginning just before uh Zak went on to um onto Ro Rogan's podcast um Donald Trump uh held a bit of a summit at maral Lago um with anybodyeverybody who's any anybody in the tech industry right after that it was very obvious I think that there was a realignment um with within the um within the tech industry of where their values now currently I think you can see me shaking my head right and it's not at the it's not at what's happened it's not at the inevitability of it all it's just at the fact that things are changing right and regardless of your political views that correlation is extremely obvious about what is happening in the US tech sectorand it's kind of scary it's kind of scary the amount of power that is available from different angles if you know what I mean especially with the Advent of AI and the the um how that can be um jumped on by any government um to their own ends so it's one to watch it's definitely one to watch and that that is a burning um that is a burning situation as we move towards um administrative Handover shall we say it that way um which is not not far away now as well so say that way but it's happening here as well I don't wantto labor the point but um I think it was just yesterday that our um Prim prime minister Kier starmer said that he was going to make the UK an AI superpower um and starting to throw money at that um except there's been quite a few Tech analysts that have come out and said well wait a minute you don't even have the power infrastructure to support what you're talking about um have you really thought this through I do think that it was an extremely flippant media bite comment that that like we're going touse AI to power Great Britain or the United Kingdom you know it's one of those things that you just say and you're like oh right how should I actually do that so again that we could probably go for a different segment right but we're we're right up on time now George so any sort of closing thoughts there or are we in a good place with with our observations um really just uh watch how the political Sands are moving over time um and how they're moving the tech industry because the tech industry is very much going to beinfluenced by politics over the next couple of years well it's fascinating stuff thank you mate really appreciate that that was um that was the data fix segment of our show here and often I don't know what he's going to talk about until we get on the show so beate it man whatever is happening in the news but it's always bang up to date yeah we we are here but although you know the data fix is always going to be here that's an omn presentent but not everything is omnipresent including our guest today solet's get them on um we're delighted to have um to have Ben here let's see if we can get all the tech working hey Ben how are you hey Ben hello hello we've got I think we've got you down here right yeah yes yeah down left and right boom boom how are you my friend I well I'm well how about yourself it's good stuff um no it's great um like I think we've got a bit of time zone adjustment for you right because I don't know what time it must be are you on on West Coast I'm on mountain time so it's alittle past eight yeah okay not not too bad not too bad well look it's it's great to have you here and um you know I was it was Fab to actually chat to you a few weeks ago as well and to to talk and a lot we're going to talk about a lot of things right but why don't you just give like a pretty short intro about yourself and just just say hello and we'll we'll jump into the the recording here and let everyone know what we've been talking about yeah yeah of course uh hey there everyone my name is Ben Rogan aka theSeattle data guy uh I think a lot of people know me because I put out uh ton of content on data engineering data infrastructure um you can often find it either on substack here or on YouTube but um in my day-to-day I consult and help companies set up their data uh data infrastructure and kind of figure out how to you know get uh the full value from data fix whatever processes might be uh needing to get fixed so you can do more than just have a bunch of cool tools but you can also you know actually change how the business operatesand data engineering means a lot of things nowadays Ben right and I think that we're going to we're going to talk about some of those things during the during the call as well so um for for everyone that's for everyone that's here watching live thank you very much and we're going to we're going to run our interview now we are all like right here in the background and we'll be taking questions like live from the chat at the end as well if you have any so feel free to pop something in the chat um andenjoy George anything from you before we head into the interview just I'm looking forward this I think it was three or four weeks ago that we recorded this um and you know I I do often say this is one I'm looking forward to but we don't often get into the weeds and into the the sort of deep down um technical details um and I think we did here um and I think that's super important in the world of data engineering because we all we can see the results but we need to understand what is powering theseresults so for that this was a really interesting one for me anyway and I I think mentioned this before um on the pre-show but um I think the most important thing of this whole interview is my continuity um I seem to be wearing the same thing that I did on the day of the interview so that says one of two things about me and I'll let you decide what those things are uh so we're going to run now and we'll see you guys in about half an hour um Luke enjoy the interview we'll see you then see youhey everyone we're very excited for you all to be here for another episode of the data mix I'm Brian over there is that way George George and you know as mentioned earlier down the bottom we've got Ben hey Ben how are you hey hello hey Ben it's nice to have you here man how are you yeah thank you thank you it's great to be here yeah we were just talking about the like the contrasting lighting that we have going on here we're all having our own struggles at different different levels right so yeahyes yes different different sides of the world different times different SI where are you Ben uh Denver oh Denver so like so we'll have to talk about Seattle part a bit later on but yeah yeah Denver so like it's awesome to have you on the show Ben thanks for making the time and um you know just to to get us started here why don't you give us a little bit of introduction of who you are and we know where you are where you are now but who you are and what you've been doing and what all that stuff is yeah no againthanks for having me on uh my name is Ben ran a lot of people do know me as the Seattle data guy although uh currently powered by Denver um you know uh been working in data for getting close to 10 years now um initially started in healthcare so I worked at a hospital for a little bit doing kind of SQL Server ssas uh data warehousing and then work for a healthcare analytics company uh which was a lot more uh SQL Server Powershell Etc um help but like that company was focused on developing Healthcare analytics products to sell toeveryone from insurance providers to companies to help them you know improve the population Health um of their companies uh then I did Facebook for a few years uh again kind of working in data engineering and then finally I've been Consulting kind of uh the whole time so recently for the last three years I've been doing independent Consulting and then aen the content a lot of people probably know me from either my articles or my newsletter so yeah yeah you know it's a great introduction and I thinkyou mentioned one of the hollow companies their Facebook so like I I think a lot of our listeners wonder what it's like to to work at a company like Facebook I me how did what was what was your experience to to work in a Fang type of company I mean you don't have to go into it in great detail but no no wasik no it was it was a lot of I mean there's there's a lot of good things about it um you know I really enjoyed the people the people were all obviously really smart right like uh I had someoneon my team who was like a PhD in computer science um and just a whole bunch of people who are driven and trying to figure out new ways to you know better manage data better process data at least from the data engineering side if you're on the data science side you know trying to answer questions that are are very Niche um obviously there's parts of it that you feel very spoiled like uh getting breakfast lunch and dinner is nice I will say as now now as someone who works from home and has to do all that uh him himself all the time uh itdefinitely it's just like that extra thing but no it was it was a lot of fun a lot of a lot of opportunities to learn um I think it really forces you to think about the work that you're doing more because uh you're you're you're not competing but it can feel like you're competing against you know a thousand other people that are doing your job and are questionably you know just as good if not better um not not questionably they are just as good and many times much better than you at your job soyou're constantly trying to figure out okay am I doing things that are worth it um again it's not competitive per se but it's hard not to feel a little bit of competition cool do you want to talk a little bit about some of the data challenges you were wrangling with there sure I I think what's what's nice about Facebook um is you know I I've worked at a few companies now and then done plenty of Consulting and and you know the challenge you run into at a lot of companies uh is everything from havingto set up your data infrastructure yourself uh dealing with data integration right like data set a does not integrate to data set B uh very easily all all of these things and and a lot of those things don't exist at Facebook at least not in the same way uh the data is integrated you know because the way they've developed their product it's it's also kind of ground up it's digital um there's a lot of benefits that you have there versus a lot of companies have like seven Erp systems none of them talk to each otherone manages you know something on the manufacturing line one manages something in a different country allog together and you're trying to piece it all together um again not not as much of an issue and then even things like data size is is very limited in terms of like who struggles with it if you're you're if you're on a team that has big data you might have to deal with some of those issues but I'd say um a lot of teams there you know the amount of compute that Facebook has is is usuallypretty sufficient honestly a lot of the problems you run into tends to be like am I doing the right work right like am I doing work that's that's impactful and I think that's what Facebook uh really tries to do is they try to remove a lot of the technical friction uh and make it more about okay but like are you doing work that's actually helpful not are you you know solving a technical problem every once in a while you would run into uh an interesting technical issue or something I remember having to gothrough some people's pipelines um like some machine learning Engineers pipelines and it was like thousands and thousands of lines um and and and that was a little bit hairy but but overall a lot of a lot of people's workflows um I would say are in many ways uh simpler because they've got all this infrastructure that they've set up and they're constantly like they're basically constantly looking for ways to improve um um the the and reduce that technical FR friction rather than like hey can you like you know make it makeyour life harder they're trying to figure out how to make Engineers lives easier constantly yeah I would guess in a place like Facebook the the data is pretty well documented so if you're looking for a data set then you can relatively well find it yes I would I would say that it's definitely like more at least in my experience like more uh well sorry it's better documented there we go it's better documented than in most companies that I've I've worked with um some of it is because they kind of force it to happenlike there there is a forcing function where they try to like certify data sets and by doing that you have to like fill out a certain number of like column descriptions you have to describe the table have to do a few things uh in terms of like how to get that but um yeah no I would say it's it's it's definitely a better documented it always made it easy for me to kind of like jump in and like if I needed a data set I could just easily search for it and find it yeah cool I I think that opens up areally interesting discussion just about data Engineering in general right because what you're talking about is a different type of data engineering with a lot of those foundational challenges have already been resolved on a fairly long-term basis right so it's more about how can we present that and how can we build that to the user but if you're not fortunate enough to work at a Fang um in terms of that the data being struct Ed in that way data engineering means something pretty different right so likewhat can you say about that in terms of the definition of like there's kind of how you handle data engineering and there's kind of foundational data engineering so what are some of the differences there because a lot of our audience are based in the area of visualization and maybe it's a simple thing to ask but I think it's a good thing to answer actually yeah I I think in terms so you're asking like is there a difference between like foundational data engineering and what was yeah so like rabuilding the foundation is different from using the foundation to build something else right so yeah yeah I I think that's a good way of thinking about it you know there's the way I I often view um data engineering is like yeah you've got like that platform so someone has to build and manage that platform and in many companies um that is the data engineering team who either has a data platform engineer or just has to do it or has to pick a bunch of out of the- boox solutions that can can do it for them from there then you kind ofhave to build the what I consider kind of infrastructure still but it's you know that data layer that is your core data model which represents uh your company's business uh as well as you can in you know your core entities right whatever your business might be uh and that should hopefully the goal in my in my mind is generally for that to change uh at a very slow pace right like you don't want it changing so quickly that you know anyone building on top of it can't rely on it so I think that's why Iusually also uh Envision that as often infrastructure as well even it's data it's like yeah but it it should be treated in such a way that it's almost like software code where it's like it's not changing so often that people can't build on it and rely on it uh and then from there you have to have this next layer and this is where like people often I think uh Envision data uh analytics Engineers kind of step in or for most companies it's just more data Engineers but the the layer where you'retrying to build that like business value uh and trying to build those business use cases on top of it uh this might be you know directly going to an analyst or data visualization uh engineer or bi uh individual to to build on top of but that that's kind of the the vision I I have and yeah at most companies you know you're likely having to do a lot of a lot of it yourself if you're a data engineer you're you might have to spin up your own airflow instance uh you might have to uh you know set up andpick your data warehouse solution yourself so it will it will depend um the company you are at most data Engineers won't get to do it because someone will have already done it for you but like you are the person who's having to kind of manage it and if a problem occurs like in your airflow instance which is the orchestration tool it can often have issues that have to do with you know maybe how it's set up and you're going to have to go figure out why it's down and why you can't even runyour pipelines at the moment so there are a lot of little things like that that at Facebook I I didn't have to deal with at other companies I did and I think that that is one of the differences and I'm actually kind of like writing an article uh a little bit about that about like thinking about data engineering Beyond Silicon Valley because I think we read a lot of Articles uh from Silicon Valley and and and you know see the cool technology that they have and I think there are these aspects of um other issues youwill face outside of Silicon Valley that are oddly more challenging in some way um even though you know you envision like oh I want to work on these hard problems at at you know Facebook or Netflix and they are hard but there are different problems that are also hard at other companies yeah but I what I often see there is that um sometimes times the problems when you go into an organization whose business is not data data might be important to them but their main business is not data you can find um all sorts of tools cobbledtogether lots of data silos and that actually becomes a really hard problem to solve what is because you're going to look at what is the value of fixing that what tools do I have what tools do I want to invest in do I want to hire a data architect for example um do we need a data warehouse or can we just pull data right out operation systems and these ones are yeah they they're hard I've never worked for a for a Fang or any organization that has a good data structure every single organization I'veever been to has got um messy data yeah yeah and it's not say that Facebook and and things don't have messy data of course they do but it's just it's definitely different it's definitely a smoother process um like I remember when I realized I'm like all I have to do is you know for for deploying pipeline essentially just put together a pretty straightforward python script and then push it like like the company I'd worked at before that because it was kind of more of a startup like the process ofgetting code to production was was far more laborious like I like I gotta write some release notes I got to post them on this website I gota like all these little extra details um that weren't weren't there anymore yeah and I think that that's interesting you mention that because I think breath is an issue that we all face now like across all Technologies it's not just a case of okay you know python or you're good or you know you know powerbi or Tableau and you're good or you know airflow and you're good thereality is that you have to know a lot of these things and how how have you find because I think for George and I we find that has almost exploded exponentially on the layers that we work that you have to have knowledge across areas rather than specifics to be able to handle a lot of the new tech so how have you found that Trend in the data engineering world over the last couple of years especially with you know let's not say AI too much but you know probably it's impacting quite a lot of things that that we're working withnowadays so yeah no I I I think that's that's kind of an interesting part where it's like there there's this expectation that everyone is Semite technical right like everyone everyone wants or like the bosses right like VPS and directors want things to happen and a lot of those things that need to happen require some technical knowledge um or or some technical work to get done and so you know there's always been a need for you know tooling to kind of help people do it but it is becoming more prevalent right like hey it's justjust use these drag and drop tools and you're gonna be able to automate entire processes um I think you run into the same issue uh anywhere else with with that is that when you do that if you don't respect that as code like if you don't treat it like as a system that you're now developing that in the future can cause unknown issues or have unknown issues um there's a huge risk right like it's like okay I built you know 100 workflows that are all automated and I I run into companies that have this all the timeand they're like yeah but we have no idea what's going on all right and then the it team's always like frustrated because then they're like well you know the the HubSpot team developed all these automated workflows in HubSpot we don't actually know what's going on there no one's testing it um they've just built it and they're assuming it works and they're like sometimes it does sometimes it doesn't and so that is another side to it that that's always kind of hardit's always been there right I I I think the first time I I recognized this was when I was using Tableau and I wrote like an article back in like 2017 you might not it might not even even be out anymore but it was like kind of the dangers of Tableau which was just the fact that you know I was realizing that oh it's so easy to use that's kind of its own danger right like if I can just drag and drop a bunch of things anyone can do that and now um are you putting the right data out there who knows uh soyeah yeah I think I think governance is is another area that we could get into there but one thing I think what that you've really touched on in terms of like that knowledge of consultancy like already you've mentioned like airflow you've mentioned Tableau which are in quite different areas as well but also you talked about you know how do you how do you deal with like fundamental problems when when they go wrong right and at some point like at some point you're not the person that technicallydeals with those but you're the person that has to orchestrate how those problems are fixed right and that's a different skill set to have and I know that we've talked about this a little bit in the past as well but it it intrigues me how you found that part of it moving away from like a little bit less technical to maybe understanding how teams of people need to attack these types of problems so again what are you finding with that and how have you found that Journey moving that because obviously if you've comefrom you know fine type of background you'll be doing this all the time now right yeah yeah I think that's that's kind of another interesting part of the world right is you have to eventually start thinking about not just uh the technical part of of uh getting things done but also how how do you operate teams and how do you manage them but also how do you actually get things done and get the right things done um because you can know all these tools and I think this is the the tendency many of us technicalpeople do is like we we can keep scaling horizontally skill-wise it's like okay we'll just pick up another tool another data pipeline tool okay great now we know three data pipeline tools but in order to actually get things done in in organization you have to eventually start looking towards other skills you have to be like okay um how do I you know get a team to you know feel like they're attacking a problem together and and making sure they're communicating making sure they're actually pushing itacross the Finish Line right because if you let a team go at a problem more than likely what will happen is they will spiral for you know far longer than they should not in terms of like doing the wrong thing it's just it's weird how things don't get done um it's it's like the whole like like say if you have a good project manager you'll never notice if you have a bad project manager you'll notice um it's kind of what that thing so if if there's not someone to like say like okay now that we've gotten thisdone let's get you know XYZ done um and making sure that that people are very like in touch with that and kind of know what's coming next and know what they have to do next um you kind of never actually move forward so so learning those skills and being able to like communicate um you know what the is and and what needs to happen not necessarily uh to get the whole thing done but what needs to happen you know next week what needs to happen you know the next week where you're currently blocked um andhow you're planning to get unblocked all those little things really do I think add up to you being able to complete projects efficiently yeah so Ben from a um a sort of data life cycle point of view where's your sweet spot are you in the sort of um storing of data the access of data so the data pipelines are you in that last mile that you were talking about where you're doing the visualization the AI um and the sort of sexy part of it where do you set in that Continuum yeah I try to help like especially as a consultant I try to bean an an end to end you know someone that can help you set up your data infrastructure um I like to say like if you ask me to do a data visualization I will do your traditional data visualization which is like three to five metrics on top a you charge below so if you want something more than that you know happy to find someone that's specialized in it but um that that's generally what I will tell most people like I can do it but if you want something really nice you know and and you have a a vision of what you thinkthen you know we can find someone that really does that well so that's usually how I view that cool and then who tends to be your clients these days and are you are you looking at sort of Enterprise businesses startups um is there a particular industry that you're specializing in yeah definitely a lot of midsize companies um I think a lot I get a lot of projects that are either people trying to set up their data infrastructure for the F for the first time um migrating their infrastructure so they're like hey we want to go fromSQL Server to snowflake to the cloud uh or the other thing is like hey we have a problem and this could just be classic Consulting like someone set some data infrastructure up we don't know what's going on can you come in and kind of help us you know untangle this mess so those are probably the most common projects I I get coming my way yeah good and anything in the data migration kind of world then you mentioned HubSpot sort of HubSpot to Salesforce is um and the and the data behind it is that a is thatan area that you would touch uh yeah not not as much no I don't necessarily do the like uh you know customer relationships uh system kind of work uh generally I I've had a few companies that I've worked with and partnered with where they usually bring me in on the on the flip side right they're like hey we're doing the Salesforce migration can you come in and help us do the data warehouse component of it so yeah yeah and yeah and tagged on to that I'm interested to sort of continue on theskills ladder thing right of being technical to being able to sort of Team manage as well and obviously you know part of who you are being is like you you've got your outward Persona as well who's you know popular on several channels now and you got a lot of followers and you write a lot of content so like that's another thing that doesn't come naturally to a lot of people like they have a lot of knowledge stuck in here both Tech Tech and consultancy knowledge but they're either scared to show it or they don't know howso how did that was that like a natural transition for you because obviously all those followers didn't come overnight right that's a lot of lot hard work and effort and some decisions as well I guess how to split your career yeah yeah I I think you know when I decided I wanted to start doing some more Consulting I think that was part of it um another part is like I just wanted to share and create content that I didn't have uh kind of trying to get into the data world so I think those are the twothings that I really uh was focusing on was like Hey I want to get more clients and part a good way to do that is to kind of share you know who you are share your skill set be willing to kind of you know put yourself out there um and then two also just creating content that I didn't have uh you know whether it was an interview guide whether it was you know just some basic uh initial projects or something I think that that was all helpful because um what is about 10 years ago there wasn't a lot in terms oflike data engineering in like a single place there was a lot of components of it right like you could read Kimble's book for data warehousing you could you know put together all these pieces uh but there was very few pieces of content that was like here's like most of what you need to know uh for this thing called Data engineering now we have uh Joe ree's book um who's kind of put together a good high Lev view of most of what you should know but before that there really wasn't much in terms oflike a data engineering guide or or big picture kind of kind of content yeah look we'd love to get Joe on maybe maybe next year um I I had a chat with him but he just ain't got no room right now so but he's running around he's I think I just saw a post where he was like I'm putting out no content this week because I'm traveling so I I think he canned his like his regular like Monday morning um podcast as well that he did his little 15-minute catch up because it was just too much for him toto do with everything else that was happening so but I think you know related to that I think the social side of Daya engering is just like exploding right now I don't know what has happened but like my feed maybe it's just you but my feed is just blowing up right now with everything to do with data engineering and there seems to a huge appetite to to learn that stuff so where where do you think that sort of stemmed from do you think that's just a tech thing or like some of the content creators have just like gone reallyexplosive how how do you explain that yeah I think some of the content creators have gotten you know like they've gotten a lot of traction I think um You probably have a good portion of people who've become data analysts who want to get to that next phase in their career um and view data engineering as they enter right like for me you can become a high level data analyst and become super valuable to company that way but some people probably want or decided that hey you know what I think data engineering is the way that I wantto go so I think that that is some people's next phase um you know everyone everyone all there's a lot of good options I think in in the data world again you could become high level data analyst you go into Data engineering you could go into some these other and try to figure out what fits um so I think that's a little bit of it I think there is still this need for a lot of data wrangling right like everyone continually figures this out right in 2012 to like 2017 every company was like we need to hire data scientists uh andthat led to kind of this like wait before we do that we actually need to manage our data well uh and so I think that kind of created that that that cycle as well as well where people hired a buch data scientists those many of those data scientists um had to do data engineering work and weren't that happy uh some of them converted to data engineers at that point right I think Jo ree is one of those uh and then eventually you know it became more of a a career and more of an option and you know once people saw that you know it itis either something that they like doing or it pays well you know they were like okay I think I'm GNA put effort into this I think that that's manifested itself well in terms of communities and stuff like that as well and you know I know I know TFA is a big thing for you as well do you want to mention that for a bit because I've had a I've had a huge like that's had huge impact on me in terms of the people that I've met since I you know I met you and then joined TFA and that's that's made a few thingshappen so it's super cool to have that Community aspect as well so yeah yeah I mean so yeah I've got a few communities obviously one's focus on data engineering but the one you're bringing up is is focus on Consulting um and and yeah I just again same same idea where it's like I want to create things that I didn't have when I I broke into into the world of consulting which is like yeah more Community I think Consulting in particular is one of those things that's harder to teach it's notimpossible but it's harder to teach I you kind of have to go through a lot of the lessons yourself but it's nice to have other people there and and and have people that you can reach out to and ask questions about right uh in that Community you have everything from people asking about pricing they're like I don't know how to price uh these projects and you know if I charge a retain what should I charge and and I think there's a little lessons along the way like with retainers where it's likewell like for me I have certain numbers that I wouldn't do like I'm not going to do a $1,000 a month retainer it's not necessarily worth it for me to take take on something of that nature so you know helping people understand like what what problems they don't realize they're going to have uh if they try taking on 20 retainers um all that are you know very small it's like well you're going to figure out very quickly that that's all still takes up a certain amount of brain space and it's going to feel likea lot more than than you think um so so yeah I I think there's like lots of little things like that that that I hope are helpful to people yeah what is a demand like now Ben over your 10 years in uh in career have you seen demand for data increasing or for data services certainly like yours is is that on the up I I think I mean in my experience for me yes I think one of the challenges there is that it will probably feel different to different people I think because I've done put in effort putting out contentum I have a continuous demand I know that some people probably feel like hey maybe this year was more challenging you know I've talked to some Consultants who feel that and you know part of that is that if you didn't you know one some of it's just how life is I you know there's there's an aspect of luck I don't think I'll ever uh give into not knowing that luck plays a role but on the flip side I do think there's something to be said about there was a really high Point probably between 2020and 2023 is where I think you could have just been a consultant and gotten to work pretty easily it felt like um because a lot of companies had money a lot of companies were trying to do data um a lot of companies were trying to migrate uh to the cloud because of Co and and you know there was a lot of things happening and so if you started Consulting in that era you might have come out now in 2024 if you didn't create some sort of system or method to gain clients or to you know attract clients you might have come out nowfeeling a little bit um you know like it's slowed down so I I I think that's an important aspect to consider where it's like yes you know for me it's fine but I'm sure for some people if you don't have something uh you know either a network or marketing or a partnership with Avenger you're going to be having a hard time that's it's interesting that you you feel it might have slown down um in general but then I wonder if there's especially in the UK you've got some rules now where it's quitedifficult to be a sort of loan contractor consultant um and you need to put a proper business wrapper around it but what I'm seeing in uh in corporations and Enterprises um is they're on the up and up for for um hiring retained or not retained but permanent staff um in the data sphere um lots of them are doing experiments with AI um and again that's sort of putting pressure on them to get their data structures right right um and again it puts a focus on on Dat data people within the organization yeah yeah no and I'm notsaying that it has slowed down I'm just saying like for for people if you haven't I I think there definitely was more money going around where you could probably uh you know open up a pretty large consultancy in 20223 2023 uh people are obviously still investing that that's kind of almost my my point like if you've set yourself up and put yourself in a good position you're probably reaping the rewards of those uh of those project s but if you haven't um people are not I think people are beinga little more choosy and and specific still it's like yeah we got to do AI but at the same time we have a budget right and we we can't uh take out as much money as we could um in 2020 when interest was you know 1% or whatever was now now it's you know up at five six probably depending on the type of loan you're taking out I I think tagged on to that as well like I mean let's not get into the AI conversation because that's been done to death right and Ely I'll just underline that by saying that most companiesthey're just not ready for AI yet their data structures are just not ready for it and it's silly to even get in there and the companies that do start it need to start at 10% of what they're trying to do and get it right rather than trying and do it all at once so I'll just I'll just sort of close all of that off right now but I I think it I think it is fascinating in terms of like a consultancy prep and had some conversations recently with people who have asked me is this a good time to gointo consultancy and I think the answer like you said is it depends it depends who the person is it depends how hard they've been working in the background it depends if they've got something to jump to because there were times where you could jump and land pretty safely if you didn't have that Network set up but I think certainly in the UK that's a little bit harder right now but on the flip side and I think that's what George was saying if you're part of a consultancy that's already establishedyou're probably going bananas at the moment in terms of you know being able to use that effort that you have within your company I think there's no problem with that at all I still think maybe this is a UK thing right maybe it's not necessarily a relevant discussion I think it's more something to do with the contract UK Market that can be very very sensitive because of various issues that that go on there but I mean in in general would you recommend people to to like stick their toe in the water ofconsultancy and how do you how do you like make that decision how do you start what what do you think about when you're making that decision in 2024 um I trying trying to think in terms of like 2024 I guess maybe could you rephrase a question really fast yeah sure so like if you're thinking about jumping from like a permanent position to be a contractor or a consultant or to work in an area of data right now what do you think are the biggest things that you need to consider or maybe what are the areas that you would Target if it if itwere you right now yeah I think for most people if they're if they're considering making this switch um I'd I'd one don't don't jump into it too quickly I I think you can always go slowly uh that's I mean that I that's I'm biased because that's how I did it but I think I think a lot of people think that if they jump out you know they're going to have more time and if they have more time they can land more clients but there is kind of some natural cycles to to clients and and howpeople budget and things of that nature so you know building up a reputation in in the space I think is always good so you know you can always test out uh by posting on things like LinkedIn it's a great way to build your reputation and to see do people even want what I offer right because maybe you you're a specific type of consultant or a contractor and you might not know like right like you might not know if you have anything um that people want or not but if you start posting content and people are like hey yeah like I connectwith that um that I do need help with that I'd love to talk to you about whatever it is you know Tableau um airflow certain types of tooling maybe certain type of audit you think maybe uh you know you're really focused on Healthcare analytics um if you can start finding an audience I think that's always a good place to start I think that's kind of one of the benefits today of of uh using things like LinkedIn or the internet is that you you don't have to go invest a bunch of time um to figure out ifsomething can work you can you can invest a reasonable amount of time and figure out hey is there even demand for this idea or this or this you know service that I want to offer whether it's a contractor or something of that nature especially if you're early on in your career if you're later on you know maybe you've got a network maybe you've worked long enough that you can lean on that I'd say that's where a lot of Consultants I know actually focus a lot of their time is they just have a goodNetwork and that's where a lot of their work comes from um but if if you're new you probably don't have that and so you have to find a different way um to one build a network but also to to like show your skills and your expertise so I think that's what I usually tell most people as they're trying to kind of break in I think that's um you know it's quite enlightening to hear just because a lot of people that that we know and we we we see um you know watching the show maybe haven't done this before and they don'tknow where to start so to to get the advice from someone who's been there and and walked to be honest Ben you've walked the walk you've talked the talk now and I think that would be you know like a nice way to finish off here is just to you know why don't why don't you you talked about Denver you talked about Seattle but why don't you let us know a little bit about some of the ways that we can you know we can get in touch with you we can follow you online because there are quite a few so where can weget you nowadays yeah so I I try to put content out it feels like everywhere at this point so everything from YouTube uh to LinkedIn to uh substack um so uh I think those are the the three main places I put um content out uh if you want you can also join the Discord if you're interested in learning more about data engineering it's got like 8,000 people who just try to talk about learning more about data engineering or Brian as as you found out you know the the TFA has got a lot of good Consulting content onit yeah anything final from you George yeah um keep keep it short but Ben tell me what's your dream data stack I I don't know I I I try not to pick sides on on any tooling too much I think um you know I've always I've always tried to I don't think there's like a perfect set of tools um in a perfect world I think there is this future where you know you have some sort of Open Table format you know probably Iceberg since that's what everyone's going to use um that that then let you pick your youknow ice your your uh compute uh solution compute engine that you want to pick from there uh whether that's spark or or snow flake or trino or something of that nature uh and then have some level of orchestration on top of that you know um I I will always have a a soft spot for airf flow because I think that's where where I started with orchestration um so something of that nature it's it's not a perfect stack but it's like it's what works I think that's an Ideal World Companies like Facebook have made it happen wherelike again we could just pick whatever compute engine we wanted then if there was a better you know if you had a better use case for spark or a better use case for Presto you could easily just select it and it wasn't an issue um but yeah I think that's probably probably about it cool and you threw out some really cool Technologies there so thank you for that we'll not hold you to any of these though no yeah yeah no but again like it'll change in yeah five years probably so yeah yeah for for ourviewers that's not like a Ben rogojan endorsement it's not a seig guy thing it's just like we're just chatting here and look I think I think that's just like a perfect way to round off Ben it's been I like can't believe we've uh we we've locked up the timer already but it's been awesome to have you on and you know we're really looking forward to you know to having a bit of chat afterwards as well and and seeing where we go with this and we certainly wish you all thebest with with TFA and Seattle data guy and everything else that's going on so thank you so much thank you thank you thank you so much ben for sharing your 10 years worth of knowledge I appreciate it thank you all right guys thank you we will see you soon thank you very much look at that someone's got the echo someone's got the echo George G him so yeah but I'm gonna I'm just muted myself okay no worries no worries so that that I mean that was awesome Ben and um something happened like during theduring the interview right which never usually happens and that we got some pretty big breaking news I think right so let me see if I can just share my screen and maybe we can just have a very quick chat about that to start with so let's let's just talk about the first one if we can yeah yeah all right so let me just see if I can just on the Fly get there we go we're all in there so the very first one right and you were talking about iceberg right at the end of the interview there and it turns outthat that click literally today have announced that they have a acquired upsolver so what are your thoughts on that not to just throw it out to you but and I know you already scribbled the post down real quick so yeah yeah I mean I think it's interesting right like everyone I mean with the iceberg connection everyone wants to make sure they have uh you know have access to it because that's really what it's gonna be about you're going to want to make sure you have systems that can ACC either ingest into it um liketools like Estuary or or something of that nature or or have access to it from the sense of uh like your dashboard we have that I'm just gonna mute George for now okay because he's double echin keep going Ben you're good so so yeah so so they they just got Acquired and I think the I think the more interesting thing right like Rivery got acquired before the end of last year um now I'm spacing on who it was but Rivery also got uh bought out and also the the other bit of news is that SDF Labs alsoum got B out so I think what's interesting is we're seeing a lot of companies kind of consolidate uh I was talking to a VC recently and one of the things they kind of mentioned they're kind of maybe a little less interested in investing directly in uh like data infrastructure companies there's just so many right like if you ever open up Matt Turks mad landscape AP uh you can't even see the companies right it's so so tiny so I think I think we're getting to the point where some of these companies needto start consolidating so they can you know get more customers also get the benefits from all these different solutions again from clicks standpoint right like optimize probably their ability to interact with Iceberg uh and also get more customers and so I think that's what we're going to be seeing uh probably for the next little year or so I I I think we've talked about consolidation for a long time probably since Maybe 2024 a little bit of end of 2023 and I think now it's finally kind of coming into you know ittakes time for that the wave to actually happen so it's kind of kind of finally coming into its itself we'll probably see more of this I imagine over the rest of the year yeah I think you mentioned the other one there which was uh which was DBT um and let me just see if I can share that one up as well uh let's see if we can get that up to yeah DBT there we go so D DBT labs are fired SDF Labs now I got to say this is slightly out of my comfort zone right cuz I'm a little bit higher at the stack than this but againyou mentioned it but anything anything to say about about then yeah I mean so for a while there there's been uh some competition between only recently like between the the transform kind of layer uh DBT kind of set the stage somewhere in 2020 and then from there we really didn't have like DBT was kind of the tool if people were going to pick like you know SQL transform layer Tool uh since then uh a few others have kind of come about so there there's like SQL mesh uh Coes which is more of a drag anddrop tool um DBT and DBT and SDF labs are probably probably the big ones uh that people discuss so you know we've now kind of Consolidated down uh even further uh so now there's only a few really left um and and they all I think have very clear um clear what do you call it like differentiators where where some are UI based some are a little more focused on engineering some are a little more focused on the analyst like DBT uh my guess is with with SDF Labs the purchase um maybe they're looking forsome of the benefits that SDF Labs has um also just eliminates one more possible competitor in the future so and DBT is probably still sitting on a decent amount of cash I I imagine since they they had a pretty big round a few years ago yeah I mean look my my my very general take on this right is that the the the real push on AI right now means that a lot more companies are being created and successful companies are being created fast right to create a successful company you don't have to be in the ecosystem for 10 years now youcan really Blitz the market and be someone in six months time and that means there's a prerogative here on larger companies to swallow up the ones that are most important because otherwise like you just get lost in the market and you mentioned you mentioned that overview slide right where there's 2,000 companies on the same page or something and you almost can't see them anymore because there's so much just in the stck and I think that is a big big thing and if you don't go early you'regoing to get left behind I think is the other thing I was going to say as well for these bigger companies kind of and click have been quite good at doing this I know we have a little bit of Click rep on this channel right but click have been quite good at going early just picking a stack and going for it and committing and I think that's what's what's good they've sort of seen the um the the stuff that's been happening with Iceberg and the fact that it's really hitting the market in terms of apublicity standpoint as well and they've capit ized on that so I think that's that's a good move for as well um we got we got a couple of I got a good friend of mine in the chat actually I don't know if you know him Ben but a good friend of mine Dennis who I know has been boot camping on a data engineering basis for for the last couple of months now and if it doesn't include Iceberg it's a nightmare not a dream and I think we saw that at re Reinventing places like that as well right that if you'renot on the edge of this stuff you're going to get left behind so fast because there's going to be companies just implementing everything there as well I want to give George a chance to speak he's looking a bit glum there with on mute but uh let's see if we can get you in a non- echo way mate so you am I still echoing I think you're better yeah okay we're going to fix this for next time um I think I just wanted to make one one observation here and that is um what we seem to be seeing inthe data industry certainly remember we did on the data fix a couple of weeks ago uh we talked talked about data bricks and data bricks had just raised 10 billion um at a funding round um which was as much as open Ai and um anthropic put together which I thought that that blew my mind data bricks is now worth 62 billion so I think what we're seeing is um yeah yeah you're right there's um businesses are um getting off the launch pad very very quickly and growing fast but the ones there there are businesses out therethat just aren't getting funded um the smaller ones they're they're allowing them to die um and it's it's interesting that we see companies like click that have been holding on and holding on I think eight years uh since they've taken been taken private and eight years now they've been held by um Tom Bravo so it is interesting data certainly is um the industry to watch at the moment um especially for uh for m&a So Sol for for all our viewers I think you might be echoing again Georgeyou may want to just put your mute on and like for all our viewers here this was like a like a super secret added bonus it's like when you buy one of those programs right for $7 and you get an added bonus this is your added bonus is Ben Rogan commenting on live Acquisitions on the stream um so look look we really appreciate having you on Ben it's been it's been awesome both on the you know both on the recording and here as well to be able to chat to chat a little bit about this too we hope you enjoyed it too and that this wasn't thatthis is a part of your podcast marathon in the next week or so right so we're just we're just one of many oh but we they're all different topics so that's what's good that's why I enjoy it there's so many topics I enjoy talking on and it I'm it's cool that again there was there's some announcements maybe in the future uh I'll start being able to get some Scoops early and then that'll be uh even more interesting yeah so because you made that announcement on this show thatmakes us worst in line anyway yeah yes yes we'll see we'll see thanks so much mate we appreciate it we just got a couple of minutes left and we just have to get through a few things that's okay you can stay on with us Ben while we just knock these things out but again if you enjoyed our apparently we've got a new segment now where we comment on live news as it's happening in the Pod but if you enjoyed that part don't forget to give us a bit of a um a bit of a sub or a like somewhere as well scan those QRcodes thank you very much um we are super organized man we on epod episode 4 um sorry we're actually on episode four right now so there's the T first typo but for episode five man we just keep getting bigger we've got the head of um developer relations slack we've got Curtis cample um coming on as well and obviously you know slack tied in with Salesforce and Tableau as well so you know super exciting guest to have on there understand a bit more about the slack ecosystem but um I know c a littlebit I met him last year um and he's a big advocate of making technology available to everyone everywhere and he's going to talk a little bit about that story with us as well and again it's just an incredibly valuable guest for us to have on we're just we're just super humbled by everyone that is accepting us as well so looking forward to seeing Curtis and that will be exactly two weeks from today that will be on Tuesday the 28th of January we are keeping you up to date every two weeks uh with the tomix so all that's left really is for um us to all say goodbye um I'll even unmute George so he can say goodbye as well thanks again so much ben it's been totally awesome to have you here thank you everyone for listening don't forget to catch our next episode George thank you Ben I feel like I'm on the naughty step here because of my echo but uh honestly that's a really insightful uh uh podcast and a nice little bonus at the end fantastic yeah thank you thank you all super thanks everyone we'll seeyou in a couple of weeks all the bestek by [Music]