This video demonstrates how to create an OpenSearch domain from scratch and secure it within a VPC. This also provisions a Lambda and API gateway API to access the Kibana dashaboard. Git repository: github.com/lis...
Thank you for the tutorial. I am getting this error though: "errorMessage": "'NoneType' object has no attribute 'upper'", "errorType": "AttributeError", "stackTrace": [ " File \"/var/task/lambda_function.py\", line 95, in lambda_handler 'method': method.upper(), "
Hello, Looks like the event object is missing or getting passed as None. Please check your api gateway setup and try triggering a test event from api gateway.
Thanks for the video, it was useful. Would you share some details or post a video on AWS DMS target endpoint as OpenSearch creations, as it involves user role mapping.
Once the domain and the indexes are created, shouldn't that lambda proxy function be removed? I mean maybe change it in a way that it'll only expose the search api that way you can hide it behind an auth provider... otherwise the whole thing would just be publicly available for everyone. I'm not much of an expert on this that's why I'm writing this comment trying to get some guidance on the matter. What do you think?
Hello, Since the entire dashboard in exposed using api gateway, setting up proper authentication and controlling access to the api will automatically limit the access to opensearch dashboard. There are various ways to control api gateway access - docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-integrate-with-cognito.html if you are interested, here is the video about lambda authorisers - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-Q5RwxhCONy8.html
The api gateway gave me this error "Missing Authentication Token" when no query parameters were given, with any query string the gateway gave this root cause error type: "index_not_found_exception",
Hi Sapnoka, Could you please paste the URL you are trying to access? Please make sure you are hitting the right url: //_dashboards/app/dev_tools#/console If you are seeing any specific errors in cloudwatch, can you share the error?
Good solution to access opensearch dashboard. Any ways to provide authentication with the dashboard? With current configuration, it is using lambda role.
Hello, Since the entire dashboard in exposed using api gateway, setting up proper authentication and controlling access to the api will automatically limit the access to opensearch dashboard. There are various ways to control api gateway access - docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-integrate-with-cognito.html if you are interested, here is the video about lambda authorisers - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-Q5RwxhCONy8.html
pls do make a video also about creating a domain with public access and how to stream the logs to opensearch using a lambda function and visualize it in kibana dashboard, it will be more helpful if you do. Thanks in advance
Hi Benny, ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-06a3NJwM1VU.html - this video demonstrates setting up a public access domain and stream s3 data using lambda. It also shows how to access that data in kibana. You could customise it to stream log data instead of s3. Thanks.
Hello! great video and well explained, but I have the following doubt: doing this I am removing network protection from my dashboard and it becomes public, or do am I missing something? What are the benefits of doing this instead of removing the VPC from my open search domain directly? Thank you!
Hi Tobias, Thanks for your interest and to answer your question, open search domain within the VPC is still protected and you are allowing only lambda to access the open search domain. You need to add some type of auth mechanism to api gateway in order to restrict access to the dashboard. If you like, please checkout this video that explains about api gateway authorisers - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-Q5RwxhCONy8.html
@@listentolearn2363 thank you for your response! I understand your point, but if you end up protecting your api gateway with cognito for example, why don’t you do the same with open search and avoid the vpc? I’m getting into open search (I have already worked a lot with api gateway), so I would like to understand the pros and cons of the solution, or if there is any other advantage that I might not be seeing. Thank you!
VPC provides an extra layer of security. You can give this a read to understand the pros and cons - docs.aws.amazon.com/opensearch-service/latest/developerguide/vpc.html
Hey I followed all the steps but in the last step when I used my API gateway url, it says “OpenSearch Dashboards did not load properly. Check the server output for more information.”
I would suggest seeing the cloudwatch logs to check for any errors. If that looks good, then you can try enabling the api gateway logs and check for any errors there.
Same error for me to. Dashboard connected. An same error on 10 second of downloading. The error explained on stackoverflow as limitation for Lambda to 6 MB. JSON send in full size in request.
@@listentolearn2363 Errors in browser: Refused to execute inline script because it violates the following Content Security Policy directive: script-src unsafe-eval self.
@@listentolearn2363 CloudWatch. RuntimeError: Failed to post invocation response. LAMBDA_RUNTIME Failed to post handler success response. Http response code: 413.
After having a look at the API Gateway Cloudwatch logs, it seems the error is "Lambda execution failed with status 200 due to customer function error: Response payload size exceeded maximum allowed payload size"
Hi, thanks for the tutorial but I’m having trouble opening the open search dashboard after following your video. Can you provide more details on the HOST?
This is not working with OS 2.7. It keeps giving signature error "message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details. The Canonical String for this request should have been 'GET /_dashboards/app/home content-type:...... Same error with Python 3.8 or 3.11
When im trying to access api gateway endpoint I'm getting "OpenSearch Dashboards did not load properly. Check the server output for more information." And in lambda logs i see that LAMBDA_RUNTIME Failed to post handler success response. Http response code: 413, probably due to payload limit threshold ? Im wondering why it did not happen in your video ?
this is not working if you choose to user elasticsearch core in aws opensearch... you could create another video for that, maybe i just changing the py code
thanks for giving it a try. you are right, the code is specific to opensearch. however, we can get it working with elasticsearch by making few changes to the code. I will try to add it to the repo.
Hi Ketul, Could you please paste the URL you are trying to access? Please make sure you are hitting the right url: //_dashboards/app/dev_tools#/console
@@listentolearn2363 i setup everything like you explained, and after that, when i hit the API GW URL, i got the internal server error, so i thought i could test the lambda directly. When I was testing the lambda, i saw the error that i mentioned above.
ah okay, cant run standalone test on this lambda as its tied to the url.. Are you seeing any errors in cloudwatch logs when you got the internal server error?
hi, after some hit and trial. I'm getting this page and it seems like not working properly and why it is only redirecting towards only this dev tools. Can you please suggest something so its start working properly. Your help is highly appreciated OpenSearch Dashboards logo is not available Getting erros like this Expected ',' or ']' after array element in JSON at position 324
Hi Ajmal, The current python implementation only supports dev tools. If you would like to access other sections of the dashboard, please feel free to extend the code.
@@listentolearn2363 Thanks, but why opensearch page is getting broken? l was assuming we will be able to access the opensearch via this process but unable to do that. This is not the correct way to access the opensearch i beleive. Pls suggest anything else.
Hi Reshma, The region is taken from AWS session, so this should work in a different region as well. Could you please share the error that you are seeing? It would be nice if you can share the cloudwatch logs as well.
Hi Ajmal, Could you please paste the URL you are trying to access? Please make sure you are hitting the right url: //_dashboards/app/dev_tools#/console If you are seeing any specific errors, can you share the error?
Hi Alvin, What version of OpenSearch are you using? And when are you seeing this error? Is it while loading the first page or while running any specific commands?
If using a lower version is not a problem, can you try with 1.3? as I haven't tested it with 2.3 yet. I think the header is causing a problem in 2.3 but am not sure yet. see opensearch.org/docs/latest/troubleshoot/index/
@@listentolearn2363 the domain had been defined for quite some time. Was using en ec2 instance before to access it outside its vpc. So not possible to try it with 1.3. Also I tried adding the header in the lambda function, but it keeps giving internal server error. Also I’m unable to debug the lambda function at all since adding any print or log statement results in an error
I got "internal server error" because not changed every occurrence of AWS region in all listed policies first time and because of not correct URL in second time: /_dashboards/app/dev_tools#/console
