No video :(

Python AST Parsing and Custom Linting

Подписаться 230 тыс.

Просмотров 44 тыс.

50% 1

Let's write a custom linting plugin!
Linting is the process of checking your code for various stylistic and structural things that you (subjectively) want to avoid. Flake8 is a popular linter that has a lot of good default checks, but what if you want to make your own checks? In that case, you can write your own custom Flake8 plugin, which is what we do in this video! We will use the ast module to programmatically parse and examine the structure of our Python code in order to actually implement the checks.
Note: flake8 no longer supports 4 letter prefixes as of version 5, so MCOD has been changed to MC in the source code.
― mCoding with James Murphy (mcoding.io)
Source code: github.com/mCodingLLC/VideosS...
flake8: flake8.pycqa.org/en/latest/
ast docs: docs.python.org/3/library/ast...
Another great flake8 plugin tutorial: • a flake8 plugin from s...
Import loops vid: • Avoiding import loops ...
Multiple assignment vid: • Multiple Assignments i...
Testing/packaging vid: • Automated Testing in P...
SUPPORT ME ⭐
---------------------------------------------------
Patreon: / mcoding
Paypal: www.paypal.com/donate/?hosted...
Other donations: mcoding.io/donate
Top patrons and donors: Jameson, Laura M, Dragos C, Vahnekie, John Martin, Casey G
BE ACTIVE IN MY COMMUNITY 😄
---------------------------------------------------
Discord: / discord
Github: github.com/mCodingLLC/
Reddit: / mcoding
Facebook: / james.mcoding
CHAPTERS
---------------------------------------------------
0:00 Intro
0:18 What's a linter?
0:49 The goal
1:12 Don't use regex
1:58 Abstract Syntax Trees
3:31 NodeVisitor
4:42 Plugin skeleton
6:36 Our first error
7:03 No local import check
8:19 Refactoring checks
9:05 Adding options
10:13 Homework
10:51 Package your plugin

Опубликовано:

12 авг 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 67

@tokiomutex4148 2 года назад

Solving this problem with regular expressions (the Unix way) is a brilliant idea, you no longer have to worry about the original problem because you're too preoccupied with finding the right pattern.

@Alche_mist 2 года назад

Bill has a problem. Bill attempts to use regex to solve his problem. Bill has two problems.

@cryp0g00n4 2 года назад

@@Alche_mist tbh regex makes me faster in a lot of ways. I keep handy examples around after figuring them out.

@fcantil Год назад

When you're first starting out with regex, it definitely feels like adding another problem but once you start to get the hang of the basics and save those old expressions you've made, it gets easier imo

@casper64 Год назад

@@cryp0g00n4 some languages/syntaxes a regex can’t be used. Html for example

@Raja-jo5dm 2 месяца назад

he told that was nightmare!

@UnFallenRain20 2 года назад

This video is such perfect timing for me, I could watch a whole series on static analysis for python.

@ren200758 2 года назад

wow I didn't expect to see homework assignments in a youtube video. making me feel I should pay for this lesson!

@6Sloth9 2 года назад

Entertaining and informative as always. Thank you for your work.

@Mutual_Information 2 года назад

2:01 Wow!! Whenever I've applied python code to other python code, I've treated it as a string. That's awesome. I've been meaning to create a tool that'll crawl over github repos and extract code quality metrics (probably to learn which repo's are worth checking out).. this would make that task *much* easier.

@mCoding 2 года назад

This might also make your task easier!! pypi.org/project/wily/

@Mutual_Information 2 года назад

@@mCoding Never heard of this but looks helpful, thanks again!

@DilettanteProjects 2 месяца назад

I like how the use of "probably" implies that you yourself don't even know why you're doing it

@victornoagbodji 2 года назад

This is a great example for the ast module 😊

@Khushpich 2 года назад

Very informative. Thanks James.

@stifferdoroskevich1809 2 года назад

Amazing! Thanks James.

@user-me4oh3wk2v 2 года назад

Thanks! This is really helpful information.

@chongyewchang1705 2 года назад

Very well explained and exercises are cool!

@mCoding 2 года назад

Glad you like them!

@hoang-himself 2 года назад

Very good timing I am learning the same stuff but with antlr

@markcuello5 2 года назад

Yeah, thanks; I never hears of Antlr.

@lphillis1 2 года назад

YO this is great

@94grzech Год назад

good shit my man

@MrSteini124 2 года назад

Missed ya!

@mCoding 2 года назад

Glad to be back doing Python!

@VectorAmrit2 Год назад

This is what I was looking for from past 3 days. 😊

@Graham_Wideman 2 года назад

Another great topic choice James! Judging by comments, I am probably not alone in wishing for a sophisticated examination into how far you can get with a static analysis of Python code, where construction of the structure of objects is dynamic. Python does not force you to declare class or object members before you introduce them wherever you like. And the type of a variable is often not easy to determine statically, unless the code relies on some heuristics, which could be quite tricky. So how does flake8 (and its underlying tools) deal with that, or does it even attempt to? Does it have any chance of determining whether a mention of myvar.myfeld = 1 is legit, or a typo for myvar.myfield?

@mCoding 2 года назад

This is one of those times where the dynamicism of python makes it hard for tools to give good feedback. If you are adding attributes at runtime, there is no way to know with certainty whether it was a typo or not (something something halting problem...). What you can do is hold yourself to a self imposed static requirement that you will _not_ add any attrobute that is not named e.g. within dunder init of the class. Then if you see a name set outside of init you know it's wrong if it isn't listed in init. Although this kind of thing requires type information in addition to the ast if you want to handle cases outside of class methods,, so you may need to make a mypy plugin to achieve something like that.

@Graham_Wideman 2 года назад

@@mCoding Thanks for the reply. I had long ago adopted a policy of adding all attributes in dunder init, as that enables IDE to provide lookup and completion of them, and to generally maintain sanity. It seems to me that we should enable whatever static validation is feasible, and focus runtime testing on runtime (logical) errors. Indeed, type info is the next level up and Python's runtime type resolution can make that difficult, but in a large proportion of cases the lateness of type resolution is not necessary, and it could be deduced or specified in a static manner (especially with type hints).

@etopowertwon 2 года назад

Honestly, I'd use '\s+.*import' to find local imports (though I never write function-level imports to begin with) Another possibility: AST is fun for writing DSLs. You can write extremely basic transpiler from subset of python to c#/c++ in just an evening (especially if you are not afraid of litterling parenthesis that even lisper would blush).

@KASANITEJ Год назад

'\s+.*import' this is not a good idea as this will match commented imports or a string that has import in it.

@etopowertwon Год назад

@@KASANITEJ Oh no, I will have to manually skip whole 0-2 results (most likely zero).

@lemna9138 2 года назад

Is there any reason past style to put the check function in a class instead of making them a function of the class that goes through the ast?

@mCoding 2 года назад

This was just a logical separation of concerns. Imagine you have 10s or 100s of custom checks. You won't want 100s of functions in your single NodeVisitor class, you will want to break it up into individual units that each do one check. Some checks may also be more complex and require their own helper functions, so using a class makes it easy to tell which helper functions go with which checks.

@vsolyomi Год назад

It makes me think - is it possible to bind an ast to runtime? Like - my dream is a function that prints it's source file name and line number when executed. Maybe it's not to do with ASTs... I'm not sure how to approach it.

@Mertly Год назад

I feel smarter with every video and dummer with everything I still don't know. 1010/1010

@andidomi4335 2 года назад

Would it be possible to setup pycharm to also use this custom linting checks as part of it's intelisense?

@mCoding 2 года назад

Not that I'm aware of, but maybe by writing a plugin.

@JohnZakaria 2 года назад

Is it possible to replace the ast? For example rename all the functions in a file. Or AST to code.

@mCoding 2 года назад

Yes you can use ast.NodeTransformer to modify the AST, combined with ast.unparse() to get back to text. Note that ast.unparse(ast.parse(...)) does not necessarily give back the original code though, and you may want to run it through black afterwards to normalize the code style. You can also use a library like rope.

@lawrencedoliveiro9104 Год назад

I have used this to implement a syntax-level macro facility. I was looking for a way to provide both blocking versions of API calls and nonblocking ones that used async/await. The code ends up being 90% identical, but forcing non-async clients to call async functions is fiddly, and trying to use “await” in a non-coroutine function is a syntax error. ASTs provided the answer. For a nontrivial example of this in action, see my “Seaskirt” wrapper for the Asterisk telephony engine.

@JohnZakaria Год назад

@@lawrencedoliveiro9104 I wanted to do exactly the same as you. Would you share your code (even if it is broken)

@zarifatai 2 года назад

Hi James, thanks for the video. I'm having troubles running the flake8_mcoding.py file. I get the following error. Does this have to do with my Python version? I have 3.8.10 installed. line 20, in LocalImportsNotAllowed def check(cls, node: ast.FunctionDef, errors: list[Flake8ASTErrorInfo]) -> None: TypeError: 'type' object is not subscriptable

@mCoding 2 года назад

Yes 3.9 is required for the type hints. You can delete all the type hints that use [], or upgrade to 3.9 (or even 3.10 since that is the current version). Sorry for this inconvenience!

@MrRyanroberson1 2 года назад

you can also do: "from typing import List" and replace "list[...]" with "List[...]"; 3.9 was when python decided to integrate that kind of hinting into the base code, without the need for any imported helpers

@MrRyanroberson1 2 года назад

for imports specifically, i think it would be easy enough to just... "any time 'from' or 'import' are indented", but i'm sure advanced code has things like conditional imports and whatnot that are indented, so yeah seems you're right overall

@sagarbhatia7598 8 месяцев назад

Hello. I am working on research in Change Impact Analysis (CIA). If I compare AST of 2 different python file versions? Will this tool be useful for CIA ?

@bigsmoke6414 Год назад

tbh, i would have just allowed imports to only be at the very beginning of the files (excluding all comments). That way local imports are illegal, because no non comments are allowed before them and all of this is solved, but the ast is still very interesting

@kurtmayer2041 2 года назад

funnily enough, i wasn't able to figure out how to do the eval in a way that would *not* find the "sneakily assign eval to something else and use that" (i actually just visit all the names and if they're eval, i yield an error)

@sadhlife 2 года назад

that wasn't the edge case, it was globals()['eval'](...)

@kellymoses8566 2 года назад

I really wish version control systems like Git worked on the AST of the code instead of the test. It would have a lot of advantages.

@mthf5839 2 года назад

The problem with AST is that you loose all formatting, like whitespace, comments, etc. You could use the CST, but I do not the advantage there (and actually, I think it should be possible to do that with git hooks and some hacks).

@tetraxile 2 года назад

discord gang

@angryman9333 Год назад

Do Javascript AST baby

@sadhlife 2 года назад

I just wrote a 10k word article on writing your own ast based linter from scratch, coincidence? :P

@mCoding 2 года назад

I'm not familiar with your article, but feel free to post a link, I'm sure some may find it a useful resource.

@sadhlife 2 года назад

the link disappears when I post

@arisweedler4703 2 года назад

@@sadhlife what can I google to find your article?

@sadhlife 2 года назад

@@arisweedler4703 learn Python ASTs by building your own linter

@niazhimselfangels 2 года назад

Fabulous article! Thanks for going so thoroughly, and please keep the posts coming! 😍

@markcuello5 2 года назад

Help me

@DJStompZone Месяц назад

r"^\s+?(?!#)(import\s+\w+|from\s+\w+\s+import\s+\w+)" I get what you're saying, but finding local imports with regex might not have been the best example to use, since that one would actually be pretty easy. You would just need to match "(from ...) import ...", preceded by at least one whitespace character. EZPZ

@mCoding Месяц назад

Your regex matches a multiline string that contains a local import! Like """ def f(): import x """ Try again!