If you know for sure that a getter is all you'll ever need from Config, the functional approach is fine. But say in the future your need Experiment to have the ability to fetch nested items, or to modify the config. The Protocol can easily be extended to support these features. But for the getter, the code may need serious refactoring to accommodate these.
You have revealed to me a new way to implement this pattern that I had not thought of using partial. Moreover, the explanation you make is very elegant.... I really like this channel because it talks about something different in python, a part that was not being talked about. I hope to talk about the algorithms in due course
Arjan, awesome tutorial as always. You can use CMD+D (CTRL+D on windows) to select multiple occurrences of a string in VS code. so you can modify them all at once :)
Or using the Vim emulation, use the * command to search for occurrences of the same word, use "n" and "N" to move between results, and then the dot "." command to repeat the last change. 😎
I never really understood why the naming convention for a BeautifulSoup object is 'soup' until this video. "bs" would decrease my productivity as I would always stop to giggle a bit each time I use it.
Hmm, in the XML get() method you do "if not value return default". This should probably be "if value is None return default" because value could be 0 or "".
6:43 If you want some extra fun, you can name the function `__getitem__()` and have your adapter look and behave exactly like a dictionary. That way you won't have to do `self.config.get('data_path')` but could simplify to `self.config['data_path']`. The obvious caution here is that it's not actually a dictionary. One might get away with read-only data like key->value configuration objects, but I'd not use it for mutable data structures. You might get away with implementing the dunder methods for dictionaries, but it gets messy having a thing behave like a dictionary that isn't one. I'd probably not pick an adapter pattern at all but have a Builder/Factory that re-writes the data-structure on loading it from the config file and returns for example a dataclass. `def config_from_json(filename: str) -> dict: pass` and `def config_from_xml(filename: str) -> dict: pass` etc.
Great video and I love the new set! Glad to see some more videos about design patterns :) Just a quick note: The camera in the intro is not exactly parallel to the table. The table edge on the cameras left is a tiny bit lower than on the right.
Great video! It really shows the power of adapter pattern. Keep up the good work. I love the new studio - but please don’t cycle the color of the lights, it is quite distracting…
Changing background light is moving focus from the content to it. I hope this feedback will be useful. 🤞 Besides that, very good video, keep rolling to the 💯k 💪
Experiment expects Config type. Your XMLConfig is not... Shouldn't XMLConfig inherit Config? Also, we should expect that you can't change Experiment. Eg. It's a third party module And how do you get the play button and terminal output in a vs code tab?
Config is a Protocol class and protocols in Python rely on structural typing (duck typing). This means there doesn't need to be an inheritance relationship, Python determines at runtime whether the structures of the objects match. This is different from when you use an ABC (Abstract base class). In this case, you do need to inherit. The idea of the example is that Experiment is something that you wrote, but that you can't easily change a third-party library like BeautifulSoup. Therefore, you need to write an adapter layer between BeautifulSoup and your Experiment class.
Thank you! The difference is that the goal of Strategy is to be able to change the behavior of an algorithm without having to change the code that uses the algorithm, whereas the goal of Adapter is to adapt an existing piece of code (traditionally, a class) so that it can be used with another piece of code.
Hey Arjan, i like that video as all your other stuff :) would it be wrong or "bad" style to move file open and bs instantiation in adapter XMLConfig? for e.g. i would do in main only path_to_cfg = pathlib.Path.cwd() / "config.xml" adapter = XMLConfig(path_to_config) and then adapter be a dataclass where in __post_init__ i would do with open(file... and s.bs = BeautifulSoup... and finally i would also create a JSONConfig adapter with the slight changes in regards of the json file handling
Interesting topic as always. A small suggestion: don't delete, change and then revert lines to the original ones. It was rather distracting when you changed the configuration from json to xml and back several times in the video. Maybe it's better to use a flag or a factory to switch between both cases.
Instead of using a partial function, you could also use a simple factory that returns a function. Pylance understands that better and maintains the type hints.
Informative Video. I sensesd camera angle when you are seating on couch looks different..it broke the overall flow of viewing video and was clearly noticeable. I am not sure if it was intentional.
14:44 You mentioned we change the meaning of that get method and other things that depend on beautiful soup might not work anymore. I don't understand why other things calling bs.get won't work? From XMLConfig(BeautifulSoup), even if XMLConfig implements a get(), if other code is calling BeautifulSoup's get(), that should trigger bs's implementation of get() right? BeautifulSoup's get behaviour won't be influenced by XMLConfig's get behaviour if the other code is calling get from an instance of beautifulsoup instead of XMLConfig. More generally, the child class (XMLConfig) can override a parent class (BeautifulSoup) method (get) but if the caller is calling on the parent class, it will behave as implemented in the parent class right? Other feedback: When explaining the mermaid diagram, could you link it to which files/functions in the rest of the video is the interface, operation, adapter, adaptee? It's extra cognitive load to a learner who isn't familiar with UML or adapter patterns to relate the mermaid diagram to the rest of the demo.
Came here for this. I didn't understand how the XMLConfig class could impact in any way its parent's behavior. If a user uses the XMLConfig class to try and implement BS then it's their fault.
The thing is that if you create a subclass and override a method’s implementation, polymorphism comes into play. If another method in BeautifulSoup calls its get method and you have overridden it, then the overridden version will be called due to polymorphism. Also, if you pass an XMLConfig object to something else that expects a BeautifulSoup object and that then calls the get method, the overridden version will be called also due to polymorphism. In any case, it’s a bad idea to mix behavior from the original BS class and the Adapter. Composition is a much cleaner solution that doesn’t have these problems.
@@ArjanCodes Below code block is my understanding, you seem to be saying the last print will return child too? I can't see the issue with overriding a method from the parent in the child, as long as the user of these classes knows clearly which of the 2 prints they want to call. # BeautifulSoup is parent # XMLConfig is child class parent: def get(self): return "parent" class child(parent): def get(self): return "child" print(child().get()) # returns child print(parent().get()) # returns parent, not affected by child overriding get
What I mean is this: class parent: def another_method(self): print(f"Calling self.get: {self.get()}") def get(self): return "parent" class child(parent): def get(self): return "child" print(child().another_method()) This prints ""Calling self.get: child". The problem is that another_method from the parent class doesn't know anything about there being a subclass. So it's very well possible that calling another_method results in an error, even though we didn't change that method. This is an example of a Liskov substitution principle violation. In the subclass, the contract of the get method is changed, potentially breaking other things. To go back to the BeautifulSoup case: it's possible that the get method of BeautifulSoup is called from elsewhere in the class. By overriding the get method and changing it, we introduce potential errors.
Very interesting. I would have thought/expected to have some sort of "conformity" with the adapter to take in an XML or JSON and then henceforward "normalize" the methods and values returned (e.g. key with value) vs having to change from "get" to "load" within the experiment file. I was thinking the "middle layer" (Adapter?) might possible be a function, class, or object (object was my main guess) that would be in a separate file and then would be imported into the experiment.py file. Is that a different pattern or just me creating a babbling mess of another way to do things? I am thinking of building a web scraper. Which would have two main interfaces. One for the scrapers and another for the destinations for persistence. Focusing on the location interface, e.g. AWS S3 bucket, Postres [local or AWS], or Local file directory. The user would just have to supply the "destination" in the scrape settings file (e.g. settings.py), e.g. data.destination('Postgres') and the credentials in the respective file, e.g. config.py (or Hydra, like your XML example) -- which then would handle it all "upstream"... to the scrape.py file which has a method load_data('Postgres') on the scrape object. Although, maybe that creates an unnatural depencency and should be a function that I "inject" (use) on the scrape object.
The common problem with tutorials and educational content on design patterns is usually that the problem statements are too trivial. Ending up with a `Callable` as a dependency in such a video is elegant, but the general problem statement which requires an adapter is usually a whole different beast. The adapter is usually the second step after an interface was already established and there is a clear line drawn between producers and consumers of data. The adaptees though can be systems of any size with a small adapter ensuring compatibility with the target system. To apply this to your scraper: you should think of the interfaces: 1. One for the scrapers 2. One for defining the interface for the persistence layer 3. One for defining the persistence providers You could get along with two and simply use a factory instead of a provider. With the third interface you could define and interface for types though that test for certain pre-conditions and depend on different other libraries. Especially if you use different projects for the persistence layers that can be really useful. You’d have one interface project and different persistence implementations each living in their own project simply linked by your build tool. This may seem a bit over the top but actually makes the effort pay off. Just some thoughts.
@Florian Salihovic I must say. I really appreciate your thoughts. Definitely agree on the examples used to teach (and to convince) others to use Design Patterns are often trivial examples. I think even even having better talking about "how" and "what" would be much better than the examples used. I appreciate the comments... I will need to look more into the Provider pattern. Admittedly, I've mostly just relied on fairly heavy Inheritance and Factory method. The interfaces all make sense to me. The toughest part has always been separation of concerns and actually the expression of the methods to the user via the class where they don't have to go beyond the base data structures in Python as values within the class. If curious... I think this is where I'm headed... 1. Header Attributes (needed to get data): They can't be Python Class attributes because the names have hyphens, e.g. "user-agent" and "cross-origin-resource-policy". I could probably define them as attributes w/o hyphens, then do a method to return a key-value with the hyphens in the name -- but that's extra work for little gain. So this has to be a dictionary to enable the keys to have the right names for the Header that the API is looking for. Interface is, do I leave this as an importable dictionary with defaults for user to import and then they put into the Class that requests the API data [Interface #2]. I'm leaning towards just making the dictionary a default in the Class request from the API. But also making a separate module the user can import, update the default dict and then put that into the request obj. class GetAPIData: def __init__(self, headers = None, ...) self.headers = headers if headers else DEFAULT_HEADER # imported into file So if they want a mutable version they can import the DEFAULT_HEADER, then update a Key-value, rename it to NEW_HEADER Then... data = GetAPIData(..., headers=NEW_HEADER) # 2. Data Request Object (probably the main class) e.g. GetAPIData Class GetAPIData # would be several of these, the api_params would be different for each, but same header def __init__( self, IsOnlyCurrentSeason=0, LeagueID="00", Season="2021-22", headers=None, ) -> None: self.headers = headers if headers else {} self.IsOnlyCurrentSeason = IsOnlyCurrentSeason # these are case sensitive self.LeagueID = LeagueID self.Season = Season 3. Persistence Choices, e.g. to_csv, to_json, to_sql, to_s3 These likely would be a dependency injection method of using another class PushData and within it, the user has to define the credentials in a config.py file and then can call the method on the Data object (#2). e.g. data.save_data("sql") or keep them all like Pandas methods... data.to_sql() data.to_csv() ... ... This could still be a class as there's over 100 APIs and each of them would like/need these persistence methods. I could probably inherit Pandas Dataframe and then only have to build out the AWS_S3 and GoogleDrive ones (which would be rarely used by comparison).
As usual you push me to think real hard (which is a good thing) and I'll have to watch this again. So thank you again, thumbs (#16) up great video. I usually get in a situations where I don't know whether the input would be json or xml. So how would you write this code to except either a json or a xml config file?
I've never seen pipe in type hints before. What versions of Python support that? Is it better than using Union or Optional? And doesn't Any include None already?
As far as i understand the Config class that inherits from Protocol is just for type checking? Would it be possible to to make this class not inherit from protocoll and instead make it an abstract class (inheriting from ABC with @abstractclass decorator)? If not it would be nice if you could explain why :)
Either ABC or Protocol works. However, Protocols rely on structural typing (duck typing), so there is no inheritance relationship (the XMLConfig class doesn’t inherit from the protocol).
Good video as always, but I don't think a dictionary is a good parameter to pass a configuration in a real product. But I understand why you chose a dictionary for the example. However, my suggestion would be better to choose a dataclass for the configuration. With the dictionary you always have to reckon with the fact that not all keys are available. The data class cannot be created if it is incomplete. I like to do this with a classmethod, which makes a class out of the dictionary or throws an exception, then it ensures that after construction everything needed will be there.
Any technically includes None, but it makes logical sense to explicitly differentiate between the two. Because in a natural language, "any" implies something that exists rather than nothing (which is what None is supposed to represent).
@@dennisvlahos not quite correct. 'typing.Any | None' is equivalent to 'typing.Optional[Any]=None' since python 3.10. The addition here is to take into account the default value defined as None. Note that according to PEP-0484: "When used in a type hint, the expression None is considered equivalent to type(None)"
This pattern seems silly. You need to know how the "client" uses what's being passed in order to make a usable Adapter, making the code more coupled in the end. This pattern is fine for something as trivial as the video's example, but anything else beyond a "get" is problematic. Your client needs X? Convert your Y to a real X. That's the only safe way.
What is a ‘real X’? You’re thinking too much in terms of passing data. The Adapter pattern is a class with methods (in this example there’s only a single method). It’s much more about mapping behavior than mapping data. The ‘real X’ is anything that implements the Protocol. This abstraction is what makes the pattern powerful.
@@ArjanCodes It comes down to how much control you have over the client. If you're in control of the client's code, then you're free to define it to accept something abstract via Protocol or whatnot. If you're re-using someone else's work, you likely don't have that option; you have a defined interface and have to live by it. My own experience emphasizes the latter case, which admittedly may differ from your operating assumptions in this video. I'm new to Python, so excuse my naivete: How would the example code in the video behave if a member of Experiment did something like `for key, val in self.config:` ?
I had the same question as Mark. Instead of sending the object, why not have one more method in the class that returns a dict, and pass the returned dict to the Experiment class ?
(Irony) Yeah pass a dict around that’s the « only safe » way so that a client can do what the fk it wants with it. Contracts and types are what separates order and maintainability from otherwise inextricable shit that even the culprit will not be able to read easily after the crime.
For the functional approach you could also use a closure, right? Something like def generate_xml_getter(bs): def xml_getter(key): # here we use bs normally … return xml_getter
Yeah, I have the same question. I see that there are no responses to your comment - did you have any confirmation if closure can be applied instead of partial functions?
Because that would require the class to care about a lot of things. It would be better to define the config as class and have different providers which take care of the correct loading and mapping of fields.
