'You Build It, YOU Run It!' For Continuous Delivery

Подписаться 203 тыс.

Просмотров 18 тыс.

50% 1

You build it, you run it, is a founding idea in DevOps, but what does it really mean, and what are the implications? Continuous Delivery is the best way that we know to build software, it is the approach that is behind many of the organisations that we think of as leaders in software development. CD is a lean approach, meaning that we try to minimise work, while maximising results. That means that working efficiently matters a lot, and a key idea in helping us to work efficiently, is to make sure that responsibility for our work is in the right place. Siloed organisations are a problem when responsibility is unclear, or in the wrong hands. So you build it you run it is a sound-byte that really captures the idea of where responsibility for the software that we create should lie - with us, the creators of the software.
In this episode, Dave Farley, author of best selling books, “Continuous Delivery” and “Modern Software Engineering” describes the practice and implications of making dev teams responsible for their software. Who is it that should get the call at 3am if something goes wrong?
-----------------------------------------------------------------------------------
⭐ PATREON:
Join the Continuous Delivery community and access extra perks & content!
JOIN HERE ➡️ bit.ly/ContinuousDeliveryPatreon
-------------------------------------------------------------------------------------
🔗 LINKS:
The Rise and Fall of DevOps, Bryan Finster ➡️ riseandfallofdevops.com/5-min...
“What is DevOps”, Donovan Brown ➡️ www.donovanbrown.com/post/wha...
“A Short History of DevOps”, Damon Edwards ➡️ • The (Short) History of...
-------------------------------------------------------------------------------------
👕 T-SHIRTS:
A fan of the T-shirts I wear in my videos? Grab your own, at reduced prices EXCLUSIVE TO CONTINUOUS DELIVERY FOLLOWERS! Get money off the already reasonably priced t-shirts!
🔗 Check out their collection HERE: bit.ly/3vTkWy3
🚨 DON'T FORGET TO USE THIS DISCOUNT CODE: ContinuousDelivery
-------------------------------------------------------------------------------------
📚 BOOKS:
📖 Dave’s NEW BOOK "Modern Software Engineering" is available as paperback, or kindle here ➡️ amzn.to/3DwdwT3
and NOW as an AUDIOBOOK available on iTunes, Amazon and Audible.
📖 The original, award-winning "Continuous Delivery" book by Dave Farley and Jez Humble ➡️ amzn.to/2WxRYmx
📖 "Continuous Delivery Pipelines" by Dave Farley
Paperback ➡️ amzn.to/3gIULlA
ebook version ➡️ leanpub.com/cd-pipelines
📖 “The DevOps Handbook”, by Gene Kim, Jez Humble, Patrick Debois & John Willis ➡️ amzn.to/2LsoPmr
NOTE: If you click on one of the Amazon Affiliate links and buy the book, Continuous Delivery Ltd. will get a small fee for the recommendation with NO increase in cost to you.
-------------------------------------------------------------------------------------
CHANNEL SPONSORS:
Equal Experts is a product software development consultancy with a network of over 1,000 experienced technology consultants globally. They increase the pace of innovation by using modern software engineering practices that embrace Continuous Delivery, Security, and Operability from the outset ➡️ bit.ly/3ASy8n0
Roost, An Ephemeral DevOps Platform, automates your DevOps pipeline. It creates ephemeral DevOps environments on-demand or based on pull requests. Roost reduces DevOps complexities and shortens release cycles with fewer engineers. ➡️ bit.ly/CD2Roost
Tricentis is an AI-powered platform helping you to deliver digital innovation faster and with less risk by providing a fundamentally better approach to test automation. Discover the power of continuous testing with Tricentis. ➡️ bit.ly/TricentisCD
TransFICC provides low-latency connectivity, automated trading workflows and e-trading systems for Fixed Income and Derivatives. TransFICC resolves the issue of market fragmentation by providing banks and asset managers with a unified low-latency, robust and scalable API, which provides connectivity to multiple trading venues while supporting numerous complex workflows across asset classes such as Rates and Credit Bonds, Repos, Mortgage-Backed Securities and Interest Rate Swaps ➡️ transficc.com
LaunchDarkly is a first-of-its-kind scalable feature management platform that allows development teams to innovate faster by transforming how software is delivered to customers. We want to show you what we're all about. Book a demo to see our platform in action! ➡️ tinyurl.com/CDLaunchDarkly

Наука

Опубликовано:

3 янв 2023

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 83

@PelFox Год назад

Wish this was true. However we build things together with bunch of contractors that eventually leave. Years down the line you are now a small team maintaining over 10 different products so it's no clear ownerships.

@DryBones111 Год назад

One thing to add to this idea is that while the expectation is for devs to have ownership over the running of their systems, there is not also the expectation to have deep knowledge of the infrastructure that runs it. It is simply infeasible to have deep knowledge across all domains. The practice of platform engineering is about solving all of the difficult and repetitive issues that all systems face (such as networking, security and observability) while exposing tools that enable devs to keep a pulse on their system. It's all about building the right abstraction to make it as easy as possible to run it, so more time can be spent developing value while still gleaning information from how the system is meeting the non-functional requirements.

@vanivari359 Год назад

what i hate about that "you build it, you run it" stuff is that it's used to put teams in charge of everything, every project gets it's own Kubernetes cluster and that's it. A couple of years ago, a developer with 1 year of experience and a lot of personality got together with a clueless sales guy and spent half a year in a pilot project to tell the customer that it's a good idea that every microservice team (usually ~5 people) is responsible for their infrastructure - because "DevOps" and "You-build-it, you-run-it". Then he used a bunch of Helm charts from the internet and set up everything - postgres, prometheus, efk stack, a distributed cache, jenkins etc. for his team and convinced the customer that he doesn't need any platform or enabling team and should not provide any guidelines or recommendations or central templates or anything. It took a year to repair the damage done by those two guys and i see similar ideas again and again. The harm and waste created by people running around with concepts they do not understand is unbelievable. Our industry is irresponsible, there are no standards somebody has to fulfill to make critical decisions and we put more and more responsibility in the hands of those people. Imagine, actual architects (the ones constructing buildings) would act the same way. The amount of people promoted to "lead architect", which struggle with concepts like thread-safety - crazy. And that's not a problem of one company, that's rampant in giant IT service provider corporations with 100k+ employees because they can only grow by hiring more and more people. Maybe i should stop taking those firefighter-assignments to analyse and fix the messes created by "self managed agile devops teams". The last one: 50 people, 7 teams and the average developer job experience was bellow 1.5 years, the lead architect had bellow 4 years. Millions of $ and after not even a year, the system was basically not maintainable anymore. What are we doing.

@mdesnica Год назад

This also reflects the general counterproductive phenomena , in our modern time of complexity in IT solutions: instead of letting developers nische down and specialize, they have to be more generalists!!! That was a bad idea even in the 90s. "Devops", together with slogans like "full stack developer" and the expectation that every developer shall be able to take a jira ticket whatsoever from the backlog (which is a total madness in its own), has made the work environment to toxic that I am not surprised people leave software developement.

@andresdigi25 Год назад

@@mdesnica Do you think this generalization and assuming that anybody can learn and do anything(learning new tech stacks, learning db, IaC, backend etc is a bad idea and goes against quality?

@mdesnica Год назад

@@andresdigi25If its expected to be learned too quickly and just "fix the Jira ticket"; then yes. My opinion. Of course a developers shall always tinker and learn tech stacks, some new language etc.

@askingalexandriaaa Год назад

This has got more to do with not being lean. The question to keep asking is “are we getting enough feedback for changes to go to production”. If every microservice/team are mandated to adopt a cluster, by definition they do not have independence. That’s “we built something and you have to use it”.

@SomeRandoInternetPoster Год назад

I agree with “you build it, you run it” intention/spirit. In experience it’s almost always a raw deal. Every developer I’ve worked with has always wanted to build great software and when we knowingly haven’t it’s almost certainly on direction of more senior management/product owners to cut certain corners. We can say it’s our job to write working software and push back, in that case reality is you can push back all you like but it goes no where if our ranked and maybe paints you as negative. Then the response is maybe move different jobs to somewhere better, which is great if the situation allows but not everyone has that luxury. If you are in a situation where you do get the time to build in robustness even then the on-call deal often works out a raw deal. On-call often in reality means an effective hourly rate pay cut. On-call you have reduced personal freedoms and accountable between specific out of business hours times but you’ll be also be on a significantly reduced rate while on call. From experience when being oncall I’d earn more money flipping burgers on minimum wage for the same inconvenience being on call and that’s being on call for billion dollar companies as a developer running our own services, never mind smaller cash strapped companies. I was originally a fan of the you build it you run it but slowly grew to avoid companies preaching that at all costs. Development is a job to pay the bills as much as some of us enjoy it and the you build it you run it often means effective pay cut, accountability for things you pushed for and got knocked back on and less time for you to do what you want to do when not in the office.

@ContinuousDelivery Год назад

Yes, in hindsight, perhaps I should have made it clear that I was not advocating for being on-call for free! When I have done it, we made being on-call voluntary and paid people extra for their time on-call.

@andresgarciagarcia5727 Год назад

"Oncall builds character" I know most devs hate being oncall, but I believe not only it's more honest than throwing stuff over a wall, but it also helps your career (if you care about your craft). The first feature I implemented in my current company took down hundreds of thousands of security cameras when I released it to prod. I think it's fair to say that's a bigger business and real-world f*-up than most people can claim. Let me tell you, it was a sobering experience, and ever since I've become a better professional. No pain, no gain. Not to mention just how burdensome it's to work with siloed, throw-over-the-wall ops teams. I remember spending days trying to root cause a bug, opening tickets, filling in forms, getting through (the wrong) logs, start over again... compare that with a service we released recently. We were able to ssh into a server, enable logging, gather tons of data, reproduce the bug locally, and deploy a patched version the following day. I know this is a hot-button issue that will trigger many but I just wanted to add my positive 2 cents.

@stephenisienyi7726 Год назад

Lol! Totally agree. I remembered being woken up around 3 AM EST in the middle of Winter 2012 by a call from our India offshore team talking about some production incident stemming from some system created by some rockstar unknown.

@marcelocueto2952 Год назад

Thanks David! your book Modern Software Engineering is a good brief to remember the important things about our work as Engineers. Cheers!

@ContinuousDelivery Год назад

Thank you!

@adambickford8720 Год назад

The problem I've seen is over time you end up with a team where nobody was involved in 'building' it, but they are saddled with running it. With a low-quality system the "devs" essentially turn into ops, so they quit. Then the next generation of "devs" get bait-and-switched, and so on.

@adambickford8720 Год назад

@@gezenews Being hired as an 'app dev' and your whole day is refreshing AMIs, updating security policies and credentials, rotating certs, fixing the ci/cd, updating vulnerable app code/dependencies, troubleshooting/restarting services, etc.

@adambickford8720 Год назад

@@gezenews You really can't. Nobody is honest about it actually being an ops/maintenance gig because its hard enough to get devs as it is. IMO its all about the middle-mangers. If everyone can do everything, they don't have to manage or coordinate. Just complain the 'devs' aren't 'agile' enough.

@gunnarthorburn1219 Год назад

Most of my time I do development, programming. But on the business side I also do business analysis/development and support. And on the technical side I also do architecture and operations. So I am quite broad. There are people who think I am over qualified for support and operations, and think I should focus more on high profile architecture and business development. However, doing the user support myself is the best input I can have for doing business development. And doing the operations myself is the best input I can have for the technical architecture. Support and operations keep me connected to reality, and not detached (as architects and business analysts can be). I think this is not a problem if you are motivated by delivering a good product and you work in an organisation that respects it. But it becomes a problem when technical architecture is seen as more prestigious and the only career way, and operations is something that is left to underpaid people to keep them underpaid.

@DirkSteinkamp Год назад

I'm in a similar situation, and I value highly the feedback I get from direct end user contact. Yet for me it somewhat often comes with bypassing "official processes", as in my case all support requests "should" go to the first-level-support first, then second-level-support, and only to me as a last resort -- yet people that were in touch with me in the past often reach out directly to me with their issues, and I respond to them directly (either taking care of the issue, or pointing them usually to the second-level support if I know they can help without my assistance). The amount of time for that is totally manageable for me, and I have the feeling that most people have most of the time a good assessment if I'm the right person to contact, and are right in skipping going through the hoops of first- and second-level-support ... Anyhow: I get my leeway in doing this despite the "official policy", but I feel like I'm kind of singular with that approach in the organization, and I'm still wondering what I can do to prevent growing the compound risk I constitute, as a side effect of that I gain much field knowledge and insights into the interrelations of my systems with others, that over time creates a truck factor of 1 ...

@DryBones111 Год назад

It sounds like you're gaining metrics on how your system is delivering on the non-functional and functional requirements right from the source. The requirements materialise at opposite ends but ultimately need to make their way into the development team.

@JonDisnard Год назад

I've always thought devops was a way to remove or reduce the relevance of ops. And yet no dev seems to want to carry the on-call pager duty. So I love the ownership model... You build it, you run it, you ultimately own it.

@krishnadaskp21 Год назад

I have worked on 3 organisations that tried to implement DevOps and in none of them the initiative came from the development side. The burden of "DevOps" was always in the Ops side. And in such setups DevOps quickly becomes Ops who works with CI/CD

@mdesnica Год назад

Would you think the same about building a house? Should the carpeters and construction do maintenance work, when the house is built? Of course, not. Other firms specialized on maintanence take over, and the carpenters continue to do what they are good at.

@cod3r1337 Год назад

Except that this analogy is completely wrong. Building software is absolutely not comparable to building a house. You build a house once, then it's done, and rarely change it in a major way after it's complete. You do maintenance because things break with use, age, weather events etc. Software is the opposite. Software is rarely ever "done", it keeps changing all the time. And its parts don't just break with use, age or weather - they become inadequate because external circumstances change. So with software maintenance is all about responding to change, while in buildings it's basically about counteracting change.

@spacemonk4874 Год назад

Even the idea of being on call is stressful. It does not help to improve the quality of work in my opinion. But if it is well compensated and if devs take turns, I think its ok in some situations.

@ntitcoder Год назад

Thanks for sharing your wisdom! I just finished reading your Modern Software Engineering book. I've always had focus on code quality before, but the book and the content in this channel really opened up a new perspective about focusing on learning especially the importance of speed of feedback. I'm considering reading your Continuous Delivery book next, but not sure if there're a lot of overlaps between the 2 books? Would you say that the CD book dives more into concrete concerns and practices regarding CD specifically? Also, do you know of any open source project which can be treated as a good example of TDD? I'd like to take a look and get a feel of what a good suite of specification looks like if possible.

@mhcbon4606 Год назад

it exists multiple kind of quality for a software. Some of them attempts to flatter your ego, others are crucially needed to deliver good softwares. People usually misunderstand the complexity of software development.

@animanaut Год назад

topic sugestion: versioning in the cicd conext. not sure if you already did that but it would be interesting

@denimunjas6831 Год назад

Where I'm at, the dev team often has to ask the product team to allocate time for critical maintenance. Very often product teams don't tolerate "I need 3 more days to write tests and go over the code again to confirm it's working fine". It also happens that some code in my ownership gets changed by someone else or gets influenced by someone else. You can also get ownership of code that was very poorly written and no time to correct this issue. All that makes me reject any ideas of on-call work. Too often, product tends to rely on it. With no on-call product knows we can't afford midnight crashes. That said I'm always a proponent of building the system in such a way that an issue can wait a few hours until working hours. We can automate lots of stuff, so let's use that automation to have the code be on-call.

@human_devops Год назад

"Keeping the System in a Releaseable State" is a good one. I don't like some of the "us vs them" narrative here though i.e. DevOps vs Continuous Delivery vs Team Topologies. It's not one thing or another. Also disagree that developers are always best placed to respond to incidents. Operations knowledge (network and infrastructure knowledge) is also vital and you can't assume that devs know what configurations their apps are actually running in. Things are more complicated than ever, labels are actually pretty unhelpful I would say. Just make sure your best people are working on your most important stuff.

@ContinuousDelivery Год назад

I don't think I said that it was "us vs them" I said that we are "allies in a common cause" and that each group, the people that approach this for a devops perspective, and the people that approached it from a CD perspective learned from the other. There is a degree to which labels don't matter, but there is also a degree to which they limit our thinking. I find the term "DevOps" a bit problematic because it is about mechanism "Dev" and "Ops" rather than what we are trying to achieve "Continuous Delivery".

@H4KnSL4K Год назад

Thanks!

@cleofaspintolimalima1627 Год назад

Wonderful

@cleofaspintolimalima1627 Год назад

Lovely

@petropzqi Год назад

I don't get it, so we don't need the SRE team anymore? Or are the sre now developers as well? Is everyone a developer as in scrum? Designers, FE developer, BE developers and now also the SRE team?

@ContinuousDelivery Год назад

Learn the seven essential techniques that you can apply to get the benefits of Continuous Delivery for your software, your team and your business. Learn at a cut price, at your pace, with video lessons from Dave Farley ➡ courses.cd.training/courses/cd-better-sw-faster

@chihotdog1554 Год назад

Hi David, I enjoyed this topic. I have one question how does the responsibility goal work with contracted services? Many service providers come in build a product or service, and by the time you can blink, they are gone, or they have swapped their resources out. I can only see this working with permanent teams?

@PavelHenkin Год назад

In my experience, the key part is 'ownership'. If you don't own it.. but have to support it, that's a recipe for failure - or at least a lot of pain.

@sneibarg Год назад

Being on call is no big deal. The challenge is convincing people who don't build technology solutions that deploying during business hours is not only possible, but preferable. Sheer insanity is deploying your software in the middle of the night only to wait until the middle of the day to realize there's a problem.

@ContinuousDelivery Год назад

💯

@PeterHitchmanYT Год назад

Too right. Just before Christmas I was involved in trying to convince such people that doing a rollout of a system between Christmas and New Year was not a good idea, because there would be no-one around to spot problems and that it would be just like doing it in non-business hours. Fortunatley it never went ahead, after a Senior VP with more experience stepped in to stop the madness.

@wewantthefunk73 Год назад

Dave, what are your thoughts on "you break it, you buy it"? Devs and the teams do their own production support. There is no separate "production support" team.

@ContinuousDelivery Год назад

I think it depends on the product. I am not suggesting that all dev teams do all of their own operations and all of their own support. This can make sense for smaller companies, startups and so on, but not for big companies. The trick, when involving other people is to keep the responsibility in the dev team, and not to attempt to defer that to other groups. So support, or ops, may offer some lines of production support, but the dev team need to be "on-call" to help when things start going outside the bounds of normal support or basic operations. I want to know how my system is doing in production. Sometimes that can be easy, sometimes it can be really hard, but I want to know.

@DryBones111 Год назад

I've seen this where I work. While the devs don't take the calls, support calls that are more complex in nature than simply repeating a part of the user guide filter down to the dev team. This gives them a good idea of where the system isn't meeting the needs of the customer. Ops tells the dev team where the system isn't meeting non-functional requirements whereas support tells the dev team where the system isn't meeting functional requirements. So having "DevOps" and "DevSupp" tightens that feedback loop to deliver better software.

@brownhorsesoftware3605 Год назад

When did enterprise devs stop being responsible for their work? When I started and worked on mainframes EVERYTHING ran in the middle of the night. If the job that was your responsibility crashed, you got in your car and drove in to fix it because other jobs were waiting and operations had to get all the output printed before a certain time. I became famous among the ops folk for refactoring away problems in programs that crashed. It was a tricky environment because it took a week to process the entire customer base. And lots of things happened on cycles of different lengths. I worked for Chicago's (and suburbs) electric utility. Same thing when I worked for a banking data center. I left that in favor of writing custom PC applications in the mid-80s. So when did things change?

@markapgar7437 Год назад

Things changed when they started siloing infrastructure away from developers. I personally saw it in the late 90's where we couldn't change anything in the db without a DBA doing it, or having only IT be allowed to deploy. Where I work now, developers are not permitted access to production systems. Can't deploy. Can't look at logs. Can't do anything without the Infrastructure team. I'm looking to change that for the next project.

@brownhorsesoftware3605 Год назад

@@markapgar7437 At that point I was doing shrink wrap language and tools in SV... I was always the most productive working with not against operations so best of luck with your efforts!

@andresdigi25 Год назад

@@markapgar7437 i am living the same. And even to try to learn a new AWS service and run a proof of concept is hard for my team. Doing a lambda function or testing a fargate solution is not easy locally. But where I work only the devops team have full access to AWS and our production environments.

@YossiZinger Год назад

Most cases, the devs getting the 3am call are in no position of making the required decisions and changes to avoid the next 3am call...

@ContinuousDelivery Год назад

True, but why is that? I'd say that it is because that many of us in this profession have abdicated responsibility for some of these things. Often we impose these restrictions on ourselves and teach poor managers bad habits. I talk about some of that in this video: "What A Software Developer's Job Is Like In 2023?" ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-gsTcKDU73WE.html

@andresdigi25 Год назад

In my company they think they do devops but they dont allow the developers to monitor the apps or even access the production environment. Only the devops team has full access to AWS and they limit the dev teams to access all the AWS services. So there are limitations to run PoCs for example

@ContinuousDelivery Год назад

As you say "they think they do DevOps" but it doesn't sound like they do to me. Sounds quite a lot more like just "Ops" rebadged. I talk about some of the misuses of DevOps, including this one, here: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-1Mcpir3Frtw.html

@desertpillar5286 Год назад

3:57 one issue I have with defining CD like this is that it implies that it should not always be released. IMO CD is not only ensuring your system is always in a releasable state, but it is that your system is always released, continuously. If the code is good enough to be checked in/merged, it is good enough to be in production.

@ContinuousDelivery Год назад

Well, in this case, I am one of the people that helped to define CD, so I think my definition stands. It is not always appropriate to push changes into production automatically. There are lots of cases where CD is used, where this would be inappropriate, or not allowed for regulatory reasons (for example). In CD circles we make a distinction between "Continuous Delivery" - working so your system is always in a relatable state, and "Continuous Deployment" - C. Delivery PLUS the automation of the decision to release. I describe that in more detail in this video: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-7SNbDWob6cI.html

@desertpillar5286 Год назад

@@ContinuousDelivery ok that’s fair. And I guess you do have a point about who decides what it means ;). And I guess you are right continuous delivery does not have to mean continuous deployment. And what I’m referring to is obviously continuous deployment. I’ve still yet to discover where C Delivery without C Deployment would make sense. In the cases I’ve come across so far where for instance breaking a connection in a trading market involves a financial penalty, then it becomes a problem of how to isolate that component from any form of deployment so that continuous deployment can be achieved. But im sure that’s not always possible to achieve.

@ContinuousDelivery Год назад

@@desertpillar5286 Don't get me wrong, C. Deployment is the natural conclusion. But I do a fair bit of work in regulated industries, some kinds of medical device require 3 months of external, 3rd party verification before release. More prosaically, the app stores for Mobile phones require a human review before they will host Apps. So it pays in both of these examples to work to keep the SW always releasable, even though you can't always release it.

@xtrailz Год назад

Yeah, no developer wants to woken up at 3am to fix problems created by another developer.

@ContinuousDelivery Год назад

...and nor does anyone else.

@MsKherson Год назад

You build it, you run it - what a funny philosophy. Imagine a car manufacturer. I guess they would luagh their head off if I tell them so. Let's say car manufactorer (Devs) works through distributors and dealers (DevOps) toward drivers (end users), and there are also service centers to handle issues (system administrators and customer support). What you suggest is to replace dealers and mechanics from the service center with manufacturer's engineers, because otherwise those engineers would not know how their car performs. I think manufacturer's engineers are involved only at initiall roll-in testing or in extremely critical rare cases after a car model goes into production. Petty much all kind of manufactures all over ther world work in a similar manner. And (you won't believe it!) they are doing well somehow. My guess is that in many cases DevOps have a technical level which does not always match the level of challanges they are facing in production. Instead of dragging Devs into that gap, they should stop vasting their time on philosophy making and get qualification required for their job.

@feuerschnitzel Год назад

In my organisation, the devops team is responsible for monitoring the systems. When issues are found, the problems are forwarded to the dev teams as user stories or for further analysis by the people configuring the systems for our customers. However, these issues are almost never picked up properly. As a result, we see a number of time bombs in the code that are getting worse over time but that are not getting fixed. The devs don't care because they don't like working on bugs (mainly non-functional) and they use the pressure from product management on features as an excuse for not working on these bugs (yes, there is almost no ownership by devs here). In the end the devops team doesn't care as well. Why report the same issues over and over again? Luckily we automated a server restart procedure which fixes issues in most cases. This is quite a dysfunctional organisation and after having put up with this and seeing no change in the past 5 years, it is time to move on to another job. I guess I will read it in the news when this company finally goes down.

@ddanielsandberg Год назад

It's a classic pattern. Management thinks they can just "sprinkle some DevOps/Agile/Scrum" on stuff and magic silver bullet things happens. Mistake no.1: Take the Ops-team, turn them into a tools team and rename them DevOps. Keep everything just as it was, just rename departments, teams, roles and processes.

@feuerschnitzel Год назад

@@ddanielsandberg in my case, I migrated from a developer/architect role to infrastructure. Monitoring is not the only thing we do and infrastructure automation and evolution with all the tooling around it is interesting work. But why have a state of the art deployment and infrastructure when the application that is running on it is crap? By the way, management just decided to increase the release cycle to even longer than what it was before. So we are going in the wrong direction.

@ddanielsandberg Год назад

@@feuerschnitzel Tell management and Developers to read "The Phoenix Project" and then "The Unicorn Project" by Gene Kim, et. al. and they will recognize themselves. If they don't care - get out of there before you end up overstressed, in a depression, developing migraines and CPPS, wanting to get out of IT and take up carpentry instead (I'm projecting here). 🙄

@banatibor83 Год назад

This video was little help to me. The model he described works for companies who develop and run software in-house to sell services. In my case we develop software to sell it, and we provide some support. Our software is deployed at around 200 different customers, we do not have access and even if we would it would be impossible to oversee all these deployments.

@ContinuousDelivery Год назад

I certainly don't assume that all software is the same, and that SAAS is the only kind of software out there, but I would disagree that this is not applicable to your situation. The implementation of how your team takes responsibility may change given circumstance like yours, but the need for your team to own responsibility doesn't. In your position, I'd be looking for ways to establish that responsibility. In fact I did this with a client last year. The release software as part of a hardware system, to remote sites, not necessarily connected to the internet. So how do you gather feedback from the use of your software? How do you figure out what your users like and what they hate? How do you respond to defects? How do you prioritise what really matters to your users? We discussed the use of a kind of series of fallbacks. If the customer was amenable, and the device was connected to the internet, we'd ask for permission to send data so that the dev team could monitor use and performance of the system. A bit like Apple or Microsoft asking for permission to "learn from your use of our software". If the system was disconnected. we discussed, again with user permission, recording data that could be collected when the system was serviced or updated. More complex, sure. Feedback is less immediate, sure, but still better than throwing the software over a wall to users that you never see or hear from, other than via a bug-report.

@Resurr3ction Год назад

I wonder how to go about this in the banking environment where the development teams and operation teams are strictly separated by policy & regulations? Literally "you can build it, but you can't run it".

@tonyjay2208 Год назад

Which regulation prevents Devs doing ops, and in which country ? Or is this just a specific company rule ?

@ivp008 Год назад

Here is an excellent talk that unpacks the segregation of duties very well: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-SczLjDOdF1E.html

@Resurr3ction Год назад

@@ivp008 Thanks, this was helpful. Amending the rules in the large organization is the main challenge here (and very nearly impossible). But one mus try.

@Resurr3ction Год назад

@@tonyjay2208 In western world it is prohibited in pretty much all financial institutions. Both in the EU and in the US. And the reason is quite simple - you cannot let just anyone at any time touch production systems that deal with client's money & sensitive info directly. Untrackable fraud could happen then, insider trading, phishing... and the bank would be liable. Hence the restrictions. Typically this is solved by having such access restricted and monitored. Traditionally only certain people could do it (ops members). Nowadays there is an additional way in form of a "break glass" system where you can get temporary access on demand even as dev. All actions are tracked, logged etc. But you must have a valid business reason, e.g. doing a release or fixing an issue. And it is temporary, e.g. for one hour. Clearly just sitting there and watching logs for prolonged periods of time does not constitute a valid case for breakglass access. But that is just one small consequence of a bigger problem. The real issue is that you as a development team cannot own let alone run the production system in regulated environment. You can mitigate it (by having both types of people on the same team) but you cannot achieve that sense of "ownership" that is crucial for this idea "you build it, you run it" because it is quite on purpose not allowed.

@tonyjay2208 Год назад

@@Resurr3ction ok, slightly different than the summary in the first statement. I work in a regulated industry and our Devs that have ops training can push to production - which is pretty much everyone. Accessing customer data (ops) is however harder than pushing to production and access elevation audited. So I don't think anyone any time would be expected, but I see no reason why every dev in an org can not also be ops - given the correct training. And of course you might require a minimum of two to deploy, in the same way as you may may require a minimum of 2 to commit to maste.

@therealjordiano Год назад

How about 'you break it, you regret it'

@user-oe7ck7yu5d Год назад

I refuse to be on call. I never negotiated my salary to be in call. To suddenly mandate that I am on call without a substantial pay raise effective immediately then I will quit. So will others. You cannot just change people's job descriptions on the fly.

@ContinuousDelivery Год назад

No one said it worked like that! I agree, you shouldn't change your working conditions on the fly. When we did this in companies that I worked in, it was voluntary, and paid for in addition to salary.

@aslkdjfzxcv9779 Год назад

"devops" guy, here. and yes, i'm here to help YOU build better software, faster.

@Ecotechnologist Год назад

How do you expect to compensate developers for on call work, or as I like to call it, modern slavery?

@1337Braz Год назад

Most big companies pay standby hours.

@ContinuousDelivery Год назад

@Ecotechnologist Год назад

@@ContinuousDelivery if only my company was doing so… in that case I agree with your video. My company put me on call and didn’t even give me a raise! And definitely doesn’t pay extra on call money.

@Gosu9765 Год назад

No, devops might mean that to you, but let's not redefine the already established definition. Just slap your own new label on this mutation like "platform engineer", but don't put another meaning to almost meaningless at this point term as it already got twisted over years thanks to similar mambo-jumbo (I believe mostly thanks to agile movement promising this new thing as a cure for all evil in the world). DevOps was about reducing how siloed the dev and ops were before and to remove the "throw request over the wall" way of working. With cloud a lot of agility is now possible - we don't have to plan capacity as much because everything we buy will be with us a long time, make purchase orders for equipment needed, spin up VMs, connect cables - a lot of "undifferentiated heavy lifting" was provided, so our responsibilities shifted, since we as a profession got time to actually learn coding instead of low level management of a ton of completely unrelated technologies and things. Now DevOps is pretty much about "automate the repetitive and build infra in a scalable way" thanks to that. On the other hand no such thing as cloud other than "no code" solutions appeared for devs. When abstractions for our jobs appeared, we got time to completely shift responsibility - devs didn't get something like that with cloud - maybe copilot will be that? It's much, much easier for them to manage infra now, but with the right tooling I would say it was similar a long time before - most of them lack knowledge about our side of the wall. Since no such help was provided to devs and they are already struggling with deadlines, tech debt, most still don't consider security of what they do (OWASP still shows the same issues and you'll still find the most basic of vulnerabilities in the wild) I would say this is bs. Most of the software around us is buggy, unfinished, unpolished - there is not that much headroom for devs to shift, while when cloud came as infra people - our availability metrics skyrocketed leaving a lot of concern for us managed. Because of that if you shift devs to ops, you'll basically just get less on the dev side - this is zero sum game in this case. For small startup - sure, you can spin up beanstalk and manage it yourself as a dev (like you could any other managed hosting before aws), but I don't see such approach coming anytime soon for big players that require a ton more tooling, devs are incapable of managing as most of it goes back to infra and ops fundamentals they lack. We might see ops people shift heavily into dev to integrate even closer with devs, devops teams will shrink as we gain more and more capability - it will go the other way around just slightly when it comes to devs becoming ops. One thing I'll definitely agree upon with you - the on call participation. The reason infra people were involved in on-call for the longest time was that because it made sense. If application shits itself we could simply do rollback, but when infra dies no dev had knowledge how to work with that - be it server going down, DNS's death once again or anything else. Since we could deal with both, it was on us. Cloud abstracts that and more importantly makes the underlying infra available to the point, when production dies randomly after hours it's pretty much only a shitty code on production nowadays and we as the infra people can't do much about it most of the time and should be not our concern. This part definitely needs to evolve and shift back to devs now, but that doesn't have anything to do with devops itself. It's just "you built it, you run it" mindset and is a separate thing. Let's not mix the two and with continuous delivery at that.

@purdysanchez Год назад

It's just a way for companies to cut back on payroll. Developers should be able to "build it and run it", but a lot of companies already want developers to do the job of project managers and business analysts in addition to building software.