Google changed how it crawls the web, so old best practices are no longer valid https://yoast.com/wordpress-robots-txt-example/
Mikamai developers blog
Google changed how it crawls the web, so old best practices are no longer valid https://yoast.com/wordpress-robots-txt-example/