❗ TEAM TRAINING: Our Team Training Offer combines practical exercises and workshops with Dave Farley, designed to enhance team skill development and achieve real improvements to working practices. Our courses are designed to encourage people to learn together, share ideas, and overcome the challenges of new ways of working. FIND OUT MORE ABOUT TEAM TRAINING HERE ➡ courses.cd.training/pages/buy-for-my-team
@@ContinuousDelivery How would you handle testing an application that mostly functions as a proxy to other services? I.e. it merely aggregates data from other services so it has no domain logic to speak of. My colleague is proposing we only use integration testing because he feels the unit tests wouldn't add anything of worth. Since even if we would add unit-tests the difference in execution time would be small because there isn't enough to test. Personally, I'm on the fence with this one. Mostly because I cannot argue for using unit-tests due to the lack of domain logic, small amount of tests the service will need, and the relatively low execution-time in both cases (no more than a few seconds).
Wow I'd thumb this comment up 100 times if I could. James Bach has a super interesting take on software testing, certainly worth having a read/watch if you are interested in building software.
Hi Trisha and Dave, when people are discussing testing I rarely hear about ISTQB which is a good body of knowledge regarding testing. I would be glad to hear your take on it 😀
Great video! Using acceptance tests to drive unit tests has worked well for me for a long time now. Could you expand more on how to effectively use approval tests, particularly in the context of avoiding the common mistake of adding tests after writing the code?
Really love your thorough explanation! This will be a very good material for providing proper understanding of test activities both to the team and management who don't grasp the understanding of test activities properly
This was a really good video, thanks Dave. You've talked about all of these things before but it was definitely worth going over them again, this was really well packaged and presented.
Pentester etc. here - I describe my work in application layer pentesting as "acceptance testing in reverse". I guess this is sort of like how antimatter is matter in a broad sense. Instead of looking to see that it does what it should I focus on whether or not an application (or API, etc.) does what it shouldn't. I thus agree largely with this taxonomy. I would like, however, to see if TDD reduces confirmation bias. Done right - without focus on implementation etc. I would hazard a guess that it does. I mention this because I have long wondered whether or not unit testing (however useful) might provoke confirmation bias.
Manual testing has its place, certainly if you are developing anything that will be subjected to humans in production. If you don't let humans test then the customer will do that for you. With predictable outcome...
Dave, this was an excellent video and very encouraging to me because it has very similar content to what I preach almost daily at my work. I've been labeled as the automated testing expert in my department - which I feel like is being labeled as an expert in breathing. I really like your list and plan on updating my testing evangelist slide decks with your ideas. One thing I'd like point out that I often start with in my slides is that I feel there are many ways to define what a "type" of test is. E.g., black box v. white box are types by methodology while Marick's quadrants are types based on function and yours are based off purpose. Perhaps I'm off course about this but I'm interested to hear your thoughts about this idea. I've been experimenting lately with in-memory test doubles as part of my acceptance tests. So far I've found that they give me lots more confidence even though it doesn't add much code coverage and doesn't really exercise more of my code than if I use stubs. I'd be interested to hear more of what you think about heavier use of test doubles.
For Acceptance Testing I tend to use "Test Doubles" at the edges of the system under test, where it breaks out and interacts with external systems, systems that are owned, developed, deployed & usually tested, by other people outside of our team. Then we can fake those interactions. I go into a lot of detail on this in my paid-for training on ATDD, courses.cd.training/pages/acceptance-testing but there is also this free video... ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-QFCHSEHgqFE.html
Thank you for this nice overview and explanation. I also like healthchecks which perform expensive consistency tests for stored data and program behavior. A great deal of problems can be discoved befor currupted data spreads throughout the system. Secondly, I am fond of defensive programming by assertions and contracts. Contracts and assertions add so much clarity to the code. They allow errors to be discovered after release, and they allow (some) integration tests to become more like end-to-end tests. They are a prestep to provable correctness, and they bring their own complexity challenges (sometimes an assertion simply cannot afford a proportional amount of time).
Types of testing: Unit Tests 2:12 Acceptance Tests 5:50 - How they can be considered as synonymous-ish with Integration Tests from the Testing Pyramid 12:58 Approval Tests 13:44 Manual Tests 15:37
From my experience * unit testing offers the most cake value when later someone modifies code. I've never understood or mastered TDD because it feels that it works against agility and requires a lot of planning. * A different/better way of understanding acceptance tests. Interesting remarks as well. Usually it's the customer who signs off but i like this one more. * From my experience the only automated tests that matter are unit and smoke tests. Smoke tests cover a lot of other tests including performance. I also like stress tests to validate and measure the tolerances of the architecture. * Most niche products can't do automation tests unfortunately. * I've always valued the idea that qa is not done by devs. Certainly for manual testing. With automation it gets a bit blurry because now the tester thinks like a programmer and brings those biases in, while the qa represents the moon power user.
In theory - having no manual regression testing is possible and would be great. In reality - businesses rarely properly invest in automated testing, meaning the test automation goes in to a backlog rather then being developed before the features are implemented like suggested with acceptance tests. Also manual regression testing is repeatable. That’s why there are manual test cases created and the important ones are placed in to regression suite once tested in a sprint.
Hi, I think you are right but wouldn't you agree there should be professional standards which should not need being discussed with other stakeholders? One of the biggest problem of our industry, I think, are programmers failing to speak up for themselves.
They aren't as repeatable as automated tests, not even close. Humans have off-days, forget things, and get distracted all the time. One of the most expensive software failures of all time, the Knight Capital disaster, where they mis-traded $2.5 billion in shares in 45 minutes and went bankrupt as a result of the poor repeatability of humans.
@@ContinuousDelivery I agree with you that manual testing isn’t a fail safe method - but I’d say it depends on the software. I see where you’re coming from for a trading application, agree that it doesn’t make sense to have manual testing cos of the amount of variability a human can’t cover. My point is that 100% automated regression test solution doesn’t apply to every piece software, for example a web app, it won’t be sustainable or reliable due to ongoing changes in the UI. Another recent example is Crowdstrike - if their release process involved one minutes of manual testing (by booting up a windows machine with the new patch), they would have caught that issue. That decision costs ~$5.4B overall for all business (according to Guardian).
But Dave aren't integration tests useful for testing integration points such as gateway classes and repositories? I always think about Integration Tests as testing my ''understanding and usage'' of external dependencies.
One way I use Integration Tests is to learn about external to the app dependencies. Such as an Rest API. Here I can add tests to assure that when my app calls that API I get an expected response from the API. It helps isolate and quickly troubleshoot when a response I'm expecting does not occur from that API.
TRIMS is a good mnemonic for value in test automation; Targeted at risk, reliable, informative, maintainable and speedy. It's very easy and common to create automated tests without much value that do not fulfill these criteria very well. The risk here is especially high when creating automated UI tests as UI's are notoriously slow and flaky in this context.
hey, i recently started a new job. i follow you for quite a while now. the tool they use is completely new to me. the setup is quite different to what i have seen before. it is a rich client application with a database in the back. the core is developed in smalltalk and is basically a black box. the part that is customizable will be done in a "basic" Derivate (vendors own implementation). and the code is stored in the database. since the tool will be used in regulatory environments, every change is tracked in the db. if you want to run the code it needs to be published. the ide and debugger is integrated in the application. (and not fun to work with) source code management is not really existing. auto completion code beautifying as well. what the team did to use some scm features is to export the routines stored in the db and checking it into a git repo. test framework? not existing. deployment to production? half manually. so i would like to start a test framework, small steps... would you have any recommendations where to start?
I think that my top priority would to get some kind of version control in-place. Assuming you are back-fill testing, then I'd start with "Automated Acceptance Testing" as it is easier to retro-fit to pre-existing code. I have a video that talks about where to start with legacy systems... ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-_gs3MjJBEfs.html
@@ContinuousDelivery thank you for your response! what is in place: export of that one table (that holds all the "basic" routines), export each line to a separate file. So for each "function" there is an export, which is automatically commited to a git. With that there is a kind of version control... unfortunately this is not done for all parts. and there are even parts that are not easy to handle. cost/benefit is not worth it for management. i will rewatch the video in that context! thank you for the suggestion about automated acceptance testing. I will try to see what is possible with the tool. You would be surprised how legacy that tool is ;-) giving the fact that they claim to be market leader ;-)
the first thought was: implement a "header" to generate a documentation "doxygen like" this was declined since it would effect all routines (and this was only in the comments!)
Thank you, Dave. As usual very dry and straight to the point. I admire your patience explaining the same things again and again and again and again, tirelessly. I'm so fed up explaining to my team mates that unit tests are way better when written first. They say: yeah, I'll do it later. And then they complain that my tickets take too long. Cause I've got to write all the missing tests first, before I can make my change. 😮
Farley here presents his "pyramid" of: unit tests, acceptance tests and tactical tests. As an example of tactical tests are approval tests. They compare subsequent runs results to first run and visual testing would be part of that. But I somehow did not pick up fully what Farley means with tactical tests. Could someone elaborate on that for me?
I believe that the idea is that there are supplementary tests that you might want to do in addition to your automated acceptance tests to help catch problems earlier and improve the overall efficiency of your process. As an example, imagine you have a service in your test environment that is unstable and it regularly makes your acceptance test run results unusable. It might be a good idea to test that the service is up or that your system is properly integrated with it before running your full test suite even though the integration is already fully tested by your acceptance tests. That way you can alert on the problem early and save the expense of running the full test suite. These tests are 'tactical' because they don't give you additional confidence in your own work or the user facing behaviour but they do provide value by saving the team time / effort
@@d3stinYwOw you are still just describing fuzzing. Maybe a terrible implementation of fuzzing would not allow you to specify a seed value but every fuzzer I have worked with has allowed this.
@@ContinuousDelivery In some kind, yes, but does it put your system under simulation of a real world workloads? I highly recommend checking out tigerbeetle simulation page, great stuff :)
I suppose that depends what you mean, if you mean testing should be as widely used as 'compiling' then I agree, but if you mean 'compiling is enough' then clearly not. I can easily write code that does the wrong things, but that still compiles.
