Don’t Do E2E Testing! 

Learn more about Acceptance Testing and Behaviour Driven Development with my on-line course "ATTD & BDD: From Stories to Executable Specifications". Find out more about my collection of Acceptance Testing courses HERE ➡ courses.cd.training/pages/about-atdd

Oh, this is bound to be a non-controversial take.

The common in-practice use of “end to end” testing is quite different today than manual testing in a staging environment. It more often than not includes automated testing rather than manual testing these days. It can still be brittle and slow but you do get quite a bit more coverage if you can stomach the wait times.

Conclusion is to do both E2E tests and tests in isolation. Not to exclude E2E testing; Many teams already do both.

What a breath of fresh air. Good thing I found your channel. I was shocked how correct my professors at college were when they told me how often companies don't follow good practices, and often end up with a jumbled up system. They killed my passion, but I am in the middle of changing jobs. I hope the new place is better.

I cannot even think about a CI/CD without some browser e2e test that is totally reliable

1000% -- It's refreshing to see someone discuss the things that have been on my mind for quite a while, and it's very difficult to get some people in organizations to let go of the false sense of security that large complex e2e or integration tests provide. They call it "quality", while it's anything but quality. What's more frustrating is that resistance is often combined with arrogance and unwillingness to learn. Actually I'd venture to say it's the opposite of quality, because you are putting confidence in a methodology that has simply proven to be inadequate for testing large complicated systems -- especially today where each component may have connections and talk to multiple others, and so on. Also, your videos have the best animations!

What I usually mean when I say E2E is a Puppeteer, Cypress, Selenium, etc type test that does things like "When Button A is Clicked by a User with Conditions X then API M is called with the correct parameters and the results will reflect the conditions." And similar things like "When API M responds with a error B we give the user a popup that says sorry the service is down." Stuff like that. For distributed systems you really need tracing of some kind plus some amount of unit testing or formal verification on the individual components.

I know a place for which E2E testing works relatively well. It works relatively well as a temporary testing suite made for a legacy monster of a system before approaching to refactor that and make it more testable and add unit tests. It gets terrible when managers see that we have test coverage now, slap themself a high five, and stop not giving time and budget to refactorize that legacy codebase.

I think the key thing is just test at many levels, and test the right things at each level. Basically the testing pyramid. Unit tests have a place, integration tests have a place, contract tests have a place and even end to end tests have a place, but don't use end to end tests to test things you can test at a lower level. For example, your end to end test might just be a simple happy path that you can use during canary or blue/green, which can do things that contract tests can't, like verify the infrastructure and services together.

I like a lot of the ideas in this channel. However, I think maybe a lot of them can only be achieved in rare perfect conditions where you have for example a.) Complete control of development related activities in an organisation b.) A team of top notch developers who all generally get on well with each other c.) People focused more on what's best for the team rather than their own career. In the real world though things are a big mess and you might have better luck herding cats.

These are wonderful ideas but every organization I've ever worked for was way too dysfunctional to ever come close to a methodical testing.

I recently started implementing Cypress into my teams app out of my own volition, and luckily this is an endeavor the company would like to move forward with. Doing it really showed me the value in system tests where we can go over every single circumstance, whereas our e2e tests only view the very limited happy path. E2E really started looking completely useless to me outside of accessibility testing and the most basic of pipeline checks for PRs.

In my experience, whenever I rely on mocking systems, the tests all pass, they are durable, everything is great... and yet in the real production system, things are breaking. The tests just adapt to the mocks over time, and so we aren't testing the real thing. Then, we have to increase complexity of the mocks, until... they are approaching the complexity of the app itself! It seems hopeless, so I just do e2e and do my best to still isolate things. Tough, very hard, automated testing is harder than building almost anything else. Still, good point of view and glad to hear it, thanks!

I like your scientific reasoning approach to testing. That being that tests are experiments that test, pun intended, our assumptions about the code.

Excellent topic but could not disagree any more completely as regards to NOT doing E2E Testing!! My main point of disagreement is your support of an alphabet soup of automated tests (Unit Test, Integration Tests, TDD, BDD, etc...) that are run by dev/qa testers, is all you need with fake/mock end points as required in order to do end to end testing. I have been developing an algorithmic options trading system for the past 5 years now and the main end to end testing I do is always from a complete business workflow point of view as submitted by the end user which I do AFTER I have completed all of those automated tests that you mentioned of which I am in complete agreement with. A single end to end test for my algo trading systems is to: create all required reference data, open a trade , monitor the trade in the app with real time market data and close the trade exactly like I would do in a production system. No faking of any api's with mocks but definitely use third party testing resources as required with the only difference being configuration requirement changes as compared to production. Any data that is like production should be clearly defined as test data and this needs to be designed in from the get go. In this way, end to end testing should be very simple and business user workflow oriented wise, completely exhaustive to verify at a minimum that the applications meets the business requirements from developer point of view. Once this is met, then you can move onto business User Acceptance testing and production Pilot testing with clear idea of successfully delivering a product that was clearly defined by the business. At least for me, E2E testing should verify your business transactions in real time and how the system manages any error situations that could arise in production.

I think this depends a lot on the domain. My background is in air traffic management systems where it is essential to test in isolation using simulators, which are themselves complex systems requiring their own testing (but can double up as a training tool for end-users). This is in no way a substitute for E2E testing though, which can, and probably will, reveal issues that escaped all prior testing. The importance of this level of testing is such that a new system will often shadow a pre-existing system for some time before becoming operational. This is obviously the most expensive time to discover an issue, so if there is any way of incorporating some degree of E2E testing earlier in the project it could be very valuable.

Dave, just wanted to note that the thumbnails are starting to be touching cringe. As much as I understand the "nature of YT and click-baits" but for god sake, there is a limit. We get here because of your competence and level of delivery - not because of weird click-baity cringe thumbnail showing some exaggerated and fake reactions/emotions.

Not what I expected to hear, I missed the fact that E2E tests are very valuable for monitoring. You don't want your users to detect that the checkout process is broken, you should detect that early with E2E tests. Run them continuously on all environments, even production, to validate business critical paths work. Of course E2E test fail from time to time. That's expected. Software updates on the users side or API changes of external systems happen all the time. It's our job to build a software that is resilient and keeps working. And if it doesn't, we should be informed swiftly to look into it. In the end, neither your unit or contract tests matter if you don't recognize that your app is broken for end-users due to a browser update or a tiny change in any external system you don't control.

Well in reality that end-to-end test is going to happen one way or another. You just have to decide whether you want to do it before the customer does.

Your description of "system tests" seems more like integration tests to me. The SUT is put in a simulated environment and expected to "integrate" (work with) the connection points presented to it. System testing -- actually testing the whole system -- should test only those aspects of the system that can't be duplicated in a test environment. A full system test basically should be a combination of functional testing (does the whole system work?) and school of hard knocks regression testing (Bob forgot to configure this once, so I guess we have to test for it now). System testing should strive to be minimal and lightweight. Too much system testing results in the bad system you describe: a ponderous final test environment that leads to too much surprise late in the testing cycle and makes releases more difficult than they should be.

you sound like you work in a perfect world where there's tons of time to write and test every aspect of the software. the rest of us who works under the gun, limited resources don't have your luxury to do that.

I like to explain the fundamental problem of testing in general to teammates with a concept of "feedback time". Aka the time between a mistake is made and the developer learns about it. And the goal is to constantly minimise this feedback time across the organisation. Contract tests and fakes at the interfaces are certainly very good tools to achieve smaller feedback time, when it comes to more complex systems.

I like the idea that we can use a range of different types of test, and through all of them we will gain confidence in the system. Basically if you only ever tested each piece of the system on its own, i'd not feel confident releasing that without at least some sort of e2e testing that tests that they all gel together properly. Still very interesting points made in the video here, I especially like the discussion about how systems a,b,c relate to each other test-wise, i suppose this is where the ideas in mockito might come from One cool thing when it comes to e2e testing is the idea that if you carry out a handful of business scenarios that together cover all of the pieces in the system, and each one has several tests to go with it, if they all pass then its unlikely there's something seriously wrong, as each piece would've had to do its job successfully when interacting with other pieces in order to produce the right result multiple times. it is always possible that you've got bugs that don't affect the result, or multiple bugs cancelling each other out, but this gets progressively less likely with each test scenario.

I think of E2E testing as "tests for tests". Integration tests are tests for unit tests, E2E tests are tests for integration and unit tests (and contract tests and load tests, if in prod, don't @ me). Users are the ultimate tests for E2E tests. Each of these stages gets less granular and more prone to mis-specification or lack of interpretability, but each should inform the layer below when a problem is detected

We replace the word « system » by « object » and it becomes the definition of a unit test. Unit tests, integration tests, acceptance tests and e2e tests all look the same. A controlled input to assess expected output. The only differences I see are the relative scale and the tools used to control inputs and assess outputs. What am I missing? We can easily imagine acceptance tests of a library/package barely being an integration test of another more complex system. The same way we could imagine unit testing a group of external systems from a system substantially bigger. Like if someone decided to test the internet. Maybe there is a system where we unit test the internet

I think the title is incorrect. There is nothing wrong with E2E testing if you automate it, and results are evaluated automatically. In my company we do massive E2E load test, to precipitate any problem much faster than any customer can. The application is subjected to this test after any kind of changes, every time. We fire 1000s of transactions simultaneously, doing every possible operation non-stop for an hour. For major changes, we run for 24 hours non-stop. If nothing breaks, nothing leaks, then it can move to release state.

Interestingly enough, this is a debate we're having at my work now. The problem is, it doesn't matter how tightly you control for you're upstream dependency's when they change on you without letting you know. This happened between 2 of our teams that resulted in a bug that reached production and caused a feedback loop when purchasing under certain circumstances and resulted in customers being charged multiple times for a single product. There's been some back and forth on how much emphasis we put on tests pull several levers to check for regressions.

It may seem like a disadvantage that your e2e testing environment is dependent on other teams software that is in flux, but if you don't test the interactions of different systems that are changing somewhere, you'll be testing it in prod. I've recently had another system my own depends on change a response to an api call in qa without ever informing us. Our e2e environment tests caught this, and we discussed it with them and got in synch again. If we'd just faked their responses out, then the issue would have appeared first in production. I'm not sure its as clear cut as this necessarily. In environments with a lot of different teams/services that all interact things will be messy any way you shake it. There can be something to be said for not trusting other teams to communicate perfectly regarding changes to their systems that effect you. I would rather things be messy in testing than have to point fingers and pull up receipts after a live incident.

For software engineering this might be the correct approach. In data engineering and big data I feel we must do end to end testing because I have to test all of the system under stress

Not sure if the recommendation is to convert e2e into Integration Testing as much as possible? Maybe better advice is to minimize e2e tests and capture max value through other forms of tests. Not doing e2e tests is bad advise.

I feel like Dave mainly describes API-based systems where it is all stays true. The UI End to End testing is only fragile because you are using unit-testing approach to write end-to-end tests. @testrigor allows you to write test 100% purely from end-user's perspective with zero reliance on details of implementation. This way some of our customers run tens of thousands of end-to-end flows with 0 flaky issues many times a day.

One director I worked for a time ago seemed to be fundamentally against anything that streamlined the development cycle, because he thought if it was too easy then his project would attract lazy developers. I found that mindset quite discouraging.

Nowadays when I think of e2e testing, I think automated.... like Cypress. So far a much more enjoyable TDD experience compared to any web unit testing I've done, so much more efficient than manual testing. Manual staging environment testing is very expensive and I've only seen it lead to arguments between the testers and devs.

I think in an ideal world, most people would prefer to use contract testing & something like WireMock to have faster running tests & to just always trust that collaborators will always work as we expect them to. Unfortunately in a more realistic world, I think E2E tests that exercise real collaborators DO provide value despite their overhead & brittleness. Almost as a form of a smoke test for critical paths before a deployment which is why it also probably makes sense to keep them low in number, so less to maintain. Most workplaces IMO would probably benefit from supplementing their unit tests with both the types of tests described in the video (higher level, black box tests with mocked collaborators) AND the more traditional E2E type to provide higher levels of confidence, versus completely ditching traditional E2E tests. So like most things in software, to do E2E or not to do E2E... it probably depends 🤷‍♂️

I've generally found that mocking system interactions for testing doubles the workload placed on developers. In a perfect world everyone shares their test cases, but even then it's much cheaper to have people write proper documentation and public interfaces - if they can't do the latter, they definitely can't do the former. Imagine if all consumers wrote tests to submit to all software libraries that they used to inform the publisher about how they were using the library.

I'm not sure how we can implement ATDD in a microservices architecture without essentially conducting end-to-end tests. If our acceptance tests require interactions between all our microservices and the frontend, doesn't that mean we are essentially performing end-to-end testing?

Great video, thank you! From my experience, all projects still are using E2E testing under smoke or regressive suites.

All kinds of tests are useful as long you know when and how to apply them. End-to-end testing might be painful but sometimes crucial and unavoidable: for example we can use end-to-end tests when developing "minimum viable prototypes" to test in practice whether the basic assumptions for the grand architecture truly hold so that we can decide early on whether its worth investing further on a system or whether it's doomed to fail. All kinds of tests have their place in the grand scheme of things.

Imho this video makes a good point about why E2E is treacherous water and leads to a slow feedback strategy, then it backpedals by introducing multiple concepts where you should use E2E, and does it in a way that only programmers (and not stakeholders) might understand. That's very high level criticism. I saw very few experts that ever made this argument within 20 minutes. But, imho, we still need a better language to adress this paradigm. I tried to convince my project leaders about 10 years ago with a similar train of thought, and I lost them at "when we do it right, we don't need E2E anymore." Because they - rightfully - argued that we will probably never make it right.

My optimal E2E test consists only of Docker container. So that you can run it locally, inside a CI/CD pipeline, maybe even in production in the same configuration. The image versions allow you to finely control the systems that are otherwise out of your control.

Your proposal is ideal but requires every layer to be on board. If you have a system of 15 layers and 4 layers have little to no tests then writing a single all layer e2e test is more cost effective than writing 4 tests and will still prove that the whole thing can be shipped. In this way full e2e are used as a stop gap. 14:40 I think you mean that the API can't be trusted and the solution is to better specify your APIs which I agree with. If you mean the network can't be trusted I don't think there's anything you can do about that.

I have to say, after listening to this and creating unit tests & integration tests for each system and E2E test to test systems working as expected. I realise that my e2e tests actually test high coupling between the systems. Always good to listen to these videos, even if I don't agree with some of the points :)

Big ball of mud, exactly what I am dealing with, it is so difficult to get a hold of the state of the application. Every test returns a slightly different result than the one before. Extremely difficult to test.

I love this channel , but i completely disagree with what you're saying at 9:35 - "we want to deploy our system" - NO , we want the whole thing to work, if we are on PROD , but the whole scenario is not working and customers cannot use it , what is the point ? Integration is the most challenging part in nowadays, but this way of thinking put on risk the whole idea - we are on prod (system B), we don't care about system A - if they are not on PROD it's not our problem.. Yes, E2E is time consuming and so on, everything that you said - i agree, but the proposed alternative is not working as well..

I agree with you, but only, if we are talking about a perfect world where the company has a very and well-developed culture of testing, QA, and CI/CD runs in a perfect scenario. For all other realities, E2E testing is a necessity. I really wish that things work so well that we could just drop using E2E tests, I truly agree with your arguments, but most companies are not yet ready.

How do we know if the dependent system breaks our contracts if you are always testing against mocks ? I’ve seen the approach of spinning up all dependent services in the docker network with testcontainers. So you always test against the real versions. If a system B that your system depends on has a new version released, you just bump the version to which you are testing against. That way you functional/acceptance tests in the pipeline will give you faster feedback. But I’ve also seen that not working if system B depends on C that depends on D… and so on. Then it either becomes a big ball of mud, or you start to mix wiremocks and real versions of services, which I still find reasonable compared to mocking everything.

If we're practising continuous deployment we ought to be able to continuously deploy a staging environment.

We are currently on the go to implement automated e2e testing. Namely to test regression of stabilized API's. I'm already concerned about the set up n clean up needed. When test app will need to need and clean up after each test. Fine when works, but introduces potential fail state during the test, which can be automatically cleaned up on hang up state, but that's additional complexity. Potentially our QA engineers might be stressed due to the complexity involved of maintenance and require additional dev time. However the promise is that regressions could be tested by pressing the button. Namely regressions are not part of the pipeline, they just help testers to run through previous behavior while allowing to test manually the new behavior. Yeah yeah, manual testing I know, it sucks, but we trying to do the best we can with what we got. However, Dave here gave me pause for thought. Not sure if I got it right regarding contract testing between systems. But.. technically it would be possible that system A exposes it's part of the contract. And system B fetches the contract input and runs the test with it, testing the output or state change? Hence each team maintain their own set of contracts and don't care about the rest. Maybe sounds over complicated too, and what Dave means is just "unit testing" in the scope of deployable unit without outward communication what so ever. But that scares me given that such tests may give false confidence, as it relies on people communication at ad hoc times. Which seems to break whole determinism promise which will only pop up in prod env. Anyone care to explain in more detail maybe? What I got wrong?

I already have been in a scenario where the tests have been skipped as a management decision, while also a lot of pressure has been put on the (mostly inexperienced) devs. Of course it failed quite spectacular. One of the things that was done while developing was to at least implement a lot of unit testing. It was crude and partially more creative than really useful, but it provided a quite high coverage around 90%. One of the worst bugs was a saving process which triggered multiple processes via the frontend, which in the end used the same storage, resulting in them overriding each other's data. The individual processes all had perfect coverage, but they haven't been tested together, which lead to the bug reaching production with quite terrible consequences. A very simple e2e would have catched this immediately (and of course we implemented one, including a others). If you build a proper pipeline it should not be a problem to implement an automated testing stage that at least simulates a user going through the critical paths to make sure that there is no oversight in the combination of some new features. If anv bug is found which is the result of one of these combinations, you should also add one more of those tests to make sure that this will never happen again.

I've developped an internal CI/CD pipeline for a few years, and h'es mistaken: CD is not what "makes sure the software is in the deliverable state", because that's actually the goal of CI. In contrast, CD takes a version that passed CI and automatically deploys it into production.

So its --- Write proper APIs, write proper clients, and proper mocks on how the system acts when sent certain inputs --- then share it with the other teams, so all one needs to do is use them properly while developing system-under-test (what to mock, and what not to mock). Please tell me if I got this right.

Basically no organizations achieve the ideal. When we evaluate idealistic recommendations like this one, we have to question: “if we don’t make it all the way, is it still valuable?” The answer here is unequivocally no. E2E tests are plainly, clearly valuable all the way until you achieve your contract perfection, which for practically everyone is never. This is not a good recommendation, because it’s all or nothing.

Thanks for the video. Does your learnings apply to Selenium testers who automate end2end scenarios. Do they need to work with devs to mock the integrating applications that form the full system?

I think we need to redefine what end-to-end tests mean.

Do you do your "contract testing" or you isolated test for your system still using a DSL?

I try to limit "end to end" testing the way you talked about it here to simple smoke tests. somewhat questionable in terms of utility at times, but they do catch if something goes super wrong in deployment. The only huge drawback is if it fails it tends to be more difficult to debug, and it almost always indicates a test gap somewhere more fine-grained.

Testing with simulators is great and the best way to test things you can't test with the real system. The way I do it is run a larger set of tests with stubs/simulators which includes negative testing that isn't possible without simulation. Then, I run a subset of tests on the real system. Ideally, we run all of the tests that are possible on the real system. One way we accomplish this is by running fewer tests every build and then each night run a full set of tests that are basically more iterations of the same scenarios.

E2E tests find incorrect assumptions between parts of the system. I would agree that you don't want more than a few E2E tests because if you haven't got a grip on interface contracts, E2E tests will be fragile and can become a serious speed bump until the revealed problems are addressed.

Thanks for this very nice video! I like the calm atmosphere!

I would also argue NOT to do unit testing. Because most of the time we're not testing algorithms and equations. We're testing if a function call other two functions. No real meaning behind the test case. So what the average developer does is just set the expected to the actual because "he wrote the code and knows that what it's supposed to do". And now if there's a bug, fixing the bug would fail the test case. The test case would lock the bug inside.

In the test case with invoice rejection, I would add an assetFail() call after the line, which is supposed to throw an exception. This way I make sure test fails if no exception is thrown.

There are so many definitions of E2E out there in the wild. For some it's simply talking to a GUI or API through some automated ways. For some it's a test across the stack for a single application, there is the application to application kind of test but it could even mean an infrastructure test. Heck even a single ping from a client to a server could qualify as an E2E scenario. I'm glad to see the rest of the world is catching up with simply being aware of one sided contract changes occurring without notice. If the contract is being defined good enough simply tracking the meta changes on the contract should be enough of an indication of things breaking somewhere on the data river across the corporate systems landscape. Sadly more often the contracts are loosely defined, so open on purpose as it is hard to make a proper contract. And thus things break on garbage being pushed down the funnel or changes made of what means what when and thus causing trouble for the handling of actual work in progress.

These are great ideas, I wish I could figure out how to implement them with the stuff my team works on. We currently put a huge amount of energy into making full test builds of the product, which our work is only a small component of, installing it on multiple platforms, and doing live testing.

Great video Dave. This confirms the way I like to do testing. Though I rarely can control how other teams do things and what they can give me, which often means that my contract testing is more or less just test against their schema. But on my side I like to expose to other teams not just my service's schema but also part of my repo where they can use my DSL I use for my tests to write some of their tests (but it rarely ever happens other teams write these tests). If teams are not cooperating with each other then in cases I can I just accept failures, and when I cannot accept failures, then I still do things in the same way but we either have no choice but to have a test server with all systems deployed to it (just to somewhat increase chances that all goes well) or ... and this is just what I wish we could do, we could use canary deployments for some types of system which could at least help us to fail fast. But usually if teams can't do things well in the first place then they cannot handle canary deployments, too, and often mentally break down with the first change to the db schema. And I am not in a position to do any grander changes to how teams work to fix things on a grander scale.

Hi Dave, suppose I am using C++ in Visual Studio. What software/tool is used for this in vs?

How do you detect that the other software doesn't satisfy the contract anymore?

If "our system" (system B) was a service that consumed messages, did some work, and published new messages, would you consider system A the consuming, system B "doing the work" and system C producing the messages? Would you mock A and C, tesitng the logic of B given that, or would A and C not being different systems in my example, and instead of considered part of system B too?

What about set of microservices owned by one team, communicating through both async messages as well as standard http request-response? Would you do only contract testing described here, treating these microservices as an external system in fact, or would you write full e2e tests, if you control the whole environment where these tests are run ?

Durability is something I found too many middy-engineers didn't seem to respect at AWS. Too many placed far too great a value in having a large number of tests, and getting there quickly by flipping/adding insignificant variables and verifying the most minor of details didn't change, without thinking of what or why, rather than adding quality tests. So much time wasted fixing tests that tested something they never should have been generating noise and wasting effort, which ultimately makes the system less reliable as it becomes a practice for engineers to save energy and effort by ignoring some failures or not really taking a closer look at them.

I like how you make people think. But generally you have too many "don't do what's mainstream" kind of clips which start to seem to me like click bait. E2E is exactly what you should do in certain conditions, and that's why most people do it. You describe some ideal situation. Yeah, of course best would be for system A's devs do own also the acceptance tests and be independent. But in reality, the stakeholder is interested in certain scenarios, not ideal modules, and the scenarios are proven to work by E2E tests.

So basically do E2E but isolated from other services. And the title is kinda clickbait.

15:55 “for each we faked it when acceptance testing … our stub that represents an external third-party system … from the perspective of our system under test it doesn't know that this is all faked” Likely my confirmation bias but I'm noticing the absence of any term starting or ending with “mock”. 17:07 “we ran our own contract tests against each end point usually the the endpoints beta test site … we sometimes had contract tests that would fail against their beta site and we'd change our integration with it once we understood the change. We'd do this by first changing our simulation in our tests [i.e. test double]“ The “drifting test doubles problem”. It may seem like a lot of additional (testing) effort but those tests capture “our understanding of how the RealThing™ works” so there is value of having the feedback that it's no longer accurate.

I understand we don't need to do a full coverage with end-to-end testing and it's sometimes logically impossible. But it has to be there even as a smoke test. Because "stubbing" and "mocking" is sometimes an oversimplification and its inputs are based on our own imagination. We still need at least several test cases with end-to-end just to make sure we didn't miss anything.

This has nothing to do with E2E, but what do you think of randomized (unit-) testing, like e.g. with Hypothesis for Python?

Nah. having some really bad dependent that may all of the sudden change in how their work. and at least checking the happy path in e2e shows to be useful. For the other things you can also have local component level tests that run on CI build. Just fake adapters to behave in whatever way u prefer and test the rest of the system.

what are the tools you use for mocking the downstream and upstream services?

I'm concerned about the simplification of having only one input and one output or just one upstream and one downstream systems. I know many examples of two-way systems. While system A may be the upstream for the request, it may be the downstream of the response. Also, a system may depend on many other systems and many more may depend on the first one. Finally, there's the state. Some systems may be stateful by design. Well, you always can re-design a stateful system as a stateless one by extracting state into another system, but then you should test that new stateful system as well, right? I'd really like to see how does contract testing work with multiple inputs, outputs, preconditions and postconditions.

“Manual testing” seems to be “actually trying the damned thing”. What’s really interesting is how many people building (and testing) a product seem to be annoyed at the idea of actually trying it.

I would say it depends. In a monolithical old style application, it is easy to demonstrate the benefits of an isolated fine grained testing like unit testing. But let's talk about distributed micro service based architecture. The micro services themselves (if they are "microservices") are small pieces of code that have a well defined , unique, and very limited business purpose. Testing those e2e could be reasonable. Now let's take it further and suppose you're using something like Flink for your streaming implementation. How do you test this kind of application? You cannot test your "streaming logic" outside of flink since it is pretty much implemented in Flink's terms. All the 'filter', 'map', 'keyBy', 'windowing'... you name it. You have no other choice but to execute a flink application e2e in order to see the results and validate them. I talked about this issue with Flink's maintainers and they have agreed that Flink's architecture does not allow you to isolate the definition of a stream from the underlying engine. This is a huge drawback of their architecture.

The main thing I disagree with is the title of the video 'Don't Do E2E Testing!' :) We lose too much confidence in the software we build when we skip E2E testing. I agree completely about the complexity, fragility, expensiveness, and slowness of E2E tests, but they're an investment. In exchange, we gain confidence that core business cases are handled as expected. We can use risk-based testing to cover only few the most important cases. The test pyramid can guide us in determining the proportions of E2E tests compared to other levels of tests. When we strategically reduce number of E2E tests, we reduce the amount of work needed to maintain these fragile and complex E2E tests and the time they take to run.

hi dave. loved your insights as always. my question to you is if we are doing the sorts of "end" 2 "end" testing so we still need to write large unit tests or are these sort of tests are enough to give us enough confidence to engage ourselves in the contunuoius delivery paradigm. its also good to note the amount of resources that we can spare and the outcome to drive given the time constraint on a project in very small teams as opposed to something you might be used to at Getco or ThoughtWorks 🤔

I work in a heavily regulated environment. While we do test every piece independently we also deploy to a test environment that is a clone of production and a group of people has to manually check to make sure everything is correct with a well documented test script to follow. With lives on the line for making a mistake the process is slow to prevent any accidents.

I've tried googling "software testing rig" and I'm surprised how little there is on this topic!

One of the major challenges with "I only test my system alone", is that we can build a system that works exactly as we expect, but it will only be AFTER deployment to production, that you realize the data you produce, which matched your expectations, did NOT match your consumer's expectations. And the systems your system depend on (system A), give you inputs that you didn't expect. You might handle those correctly, but it will turn out that your system is providing lovely, helpful error responses, and not the actual desired production state. Your layout works perfectly, IF all of the interactions and configurations between systems are assumed to be correct. Without architects that know all the systems perfectly, you're going to run into mismatches between systems when you ignore e2e.

What frustrates me endlessly is that i have to explain this again and again and again in each project i join while it seems so obvious to me. This approach works as good as it gets, if your code works against this isolated setup, it 99% of the time also works in production (after the initial setup of the interface) and you can fully test it with a behaviour driven testing approach, which also provides you a "documentation" of how your system should work as a black-box (or grey-box). It's faster, it's more reliable, it's cheaper, Frameworks like Spring and Quarkus provide all the tools, it can be executed locally... yet still people try to script giant test suits over dozens of microservices and run them all on every commit and don't invest anything in isolated TDD/BDD tests. Frustrating. Consumer driven contract tests are a nice addition, but they come with a lot of pain because the PACT framework is horrible un-intuitive and many developers have a hard time to understand the concepts.

what's the point of testing if my software works under situations I control? I want to know if my software will work out there on the wild, especially its critical business features. Of course testing that deals with coupling problems: my system *IS* coupled to other systems. We should't stop using a tool because it deals with problems it doesn't cause; only it its cons doesn't overcome its pros.

releasable or deployable state?

man his tshirts are amazing. the videos are too.

Ignoring any 'change' done by a developer you need to regularly 'daily' run automated regression end to end tests - why ? Uncontrolled Change within the environments by third parties. 1) Your users computers are being continually 'patched' for security and functional changes. (remember patch Tuesday!) 2) Your production systems are being continually 'patched' for security enhancements and network reconfigurations. 3) The browsers that your web applications run on are continually evolving functionally and updates for security (and data gathering!). Running your tests pick up this 'uncontrolled' but deliberate change to functionality and security in the environment. Our regression tests have picked up these kind of changes where our software (although unchanged) have been effected by the environment that the system runs on and the parts that run on the customers computers. This allows the developers to make fixes due to the changing environment. So if we need to run end to end just to keep up with uncontrolled change - we must certainly run end to end when developers make change.

I usually call them acceptance tests and the way I test them is by mocking external services with mocking tools like Wiremock and dockerized infrastructure in order to test the contracts as is mentioned in this video

I really like the ideas exposed here and I have thought about them myself, but I really would like to see some references or documentation where the idea is expanded or some other people discuss it; that's because I would like to share some of these ideas to my teamwork, but just saying stuff like: "ey watch, I really like these video ideas" and then sending a RU-vid link just doesn't sound very serious or professional for me

Systems testing or unit testing, isn't this assumed to have been done by competent dev teams before releasing it for testing in the true E2E? I get what you are saying as sometimes, because there is a test team involved, some developers get lazy and don't do the system test you're describing. The full E2E should be more for an integration focus in mind, to verify the intended changes and whether it is in fact the end product that the consumer wanted. Having tested software before, we still often get bogged down with issues that to my mind should have been picked up in a systems test. It just muddles the water and creates unnecessary noise.

E2E tests are pretty useful before rewrites / refactors. You scaffold your app with tests, and then you have more confidence going into chanhing parts without breaking things. Maintaining constantly e2e tests however… usually just yet another useless layer to keep engineers busy…

What if in a distributed system a certain workflow spreads across multiple services/applications? You still need a way to say if all services together do the right thing in ensemble

There's so much truth in this, but the solution evades me. The complexity can be _immense_. In my experience with e2e testing, the biggest issue I have seen, is when the _data_ changes - and this data is often _real_ data, not mocked. An example: A frontend application which makes a request to an endpoint to get data. It could be a request based on search parameters, as an example. The issue is that the data is in constant flux - it's being modified, via other software. A product availability may change, or the name of a product changes - it doesn't matter exactly _what_ that change is, other than it is _change_. With a great deal of frontend end-to-end testing, you could be using something like selenium. You want to assert, that when a search is done with specific parameters, you expect some data to be there. It's so fragile - and if you are implementing smoke tests as part of your CI/CD solution, the build fails when an assert fails - and that could be because that data isn't there _or_ it has changed. The title is no longer "ABC", it is now "XYZ" The data should _always_ be considered to be in flux. It should be expected it _can_ change. Yet this type of end-to-end testing doesn't cover that - it blindly expects, that when you do a search for a product with a unique identifier, you will _always_ get the same data back. So, to get around this, I guess you could employ techniques where the tests connect to _mock data_ - but what if the underlying systems that provide the real data have changed, but your mock data isn't updated to recognise this? - an additional data point has been added or it's type has changed - or any other manner of change has happened. That means you have to find a way to automate the update of mock data. It's a horror show. The "solution" often ends up being half-baked - you don't check for _specific_ data, just the placeholder where that data would appear - the _markup_ that surrounds it. In the case of HTML, you assert that, for example, a heading HTML tag actually has something in it - doesn't matter what, there's just _something_ there. That's fine, until the entire request fails because the product ID your tests are using is no longer there - and your software has handled this with a 404 or some other graceful fail. The end-to-end tests _don't_ know this. They timeout. The smoke tests fail, nothing gets deployed. Crazy. Horror show.

User story is end to end. Thus, test needs to be end to end. End to end of story.

Thank you Dave

Is a QA process a better idea ?

Yet another confusing video, that demonstrates how little you know about the subject. What you speak about is not E2E. You do not perform E2E testing manually on a staging environment. End to end is an automated type of test that includes UI in the test. You automate the behaviour of a person that uses the system. This test is on top of the pyramid because it tests, automatically, the entire system, including the UI. These are the tests you write with Selenium (one of the options, if you are using Java). These tests should be executed automatically by the pipeline when you do a merge into. The pipeline creates the environment, and then it runs all the tests against it: unit, integration and e2e. You usually automate those tests that are most repetitive, so you do not have to perform these manually when you do blue/green deployment. The tests you do on staging, manually, are referred as UAT and OAT. OAT especially are used to be sure that what you are about to release is indeed working as intended.