Introduction
There's a LOT of quality infosec publications lately, in blog posts, articles, videos and whitepapers. Even though I try my best, I admit it's hard for me to keep up.
Although this post is one of these publications, I already admit that the title sounds a bit confusing and maybe even scary, and I am aware of that since that's a response I got from many individuals.
So what's so special in this post that should make you want to invest 5 minutes of your precious time to read it?
I could tell you stories about research and development work that's been going on for more than a year, or mention the fact that it contains an entirely new concept in hacking, but I think I'll take the direct approach with this one:
Using a new technology that relies on black box techniques, the server-side source code of any application can be stolen, the server side memory can bemapped, and so can the data flow of server side values.
The technique is already implemented in a new tool, does not rely on anysecurity exposures, and works regardless of any existing security enhancements.
No introductions, obscure concepts or murky waters. Just facts - Get Code, Get Memory Map, No Security, Any Application.
Let's assume for a moment that the proclamations are true - so how can this information be used in penetration tests?
Although the posts in this blog were recently focused at automated scanning, it's never too late to correct the faults. Any veteran knows that the focus of any tester should always be the manual testing process, and this new information, when properly presented to a tester, can dramaticallyenhance the process of a manual penetration test:
· Optimization of the manual testing process - allow the tester to make better decisions, faster and test entry points that are more likely to be vulnerable first.
· Gaining Intel - enable the tester to understand how a certain page / entry pointbehaves under various conditions, by viewing a representation of the server-side source code, memory and cross-entry-point processes.
· Locate complex vulnerabilities - locate leads for vulnerabilities that require access tomultiple entry points, while overriding session and database values, with various perquisites and in extreme scenarios. Vulnerabilities that cannot be detected by automated tools, and are hard to locate even in manual assessments.
· Think about it… viewing the server-side source code of any component… criteria or not, it's simply awesome.
In addition, if the information can be delivered in a standard format to a black box web application scanner, it can enhance the coverage of the tool to include potential events and behaviors that only occur under extreme or rare conditions.
And what enables us to gather this information using nothing but black box techniques?
Well, I can only define it as... umm... breadcrumbs. Many tiny, seemingly useless pieces of information.
So if having the ability to gain Insight into the server side, reducing the time necessary to perform many types of tests and being able to locate vulnerabilities that nobody else can detect withoutsheer luck is of any interest to you, hang on a bit.
And just to make sure you're not losing track, here's one way to present it:
Activating Diviner's Clairvoyance feature - viewing a representation of the server side code
Viewing the Dynamic Server Memory & Processes Map Generated by Diviner
No introductions, obscure concepts or murky waters. Just facts - Get Code, Get Memory Map, No Security, Any Application.
Let's assume for a moment that the proclamations are true - so how can this information be used in penetration tests?
Although the posts in this blog were recently focused at automated scanning, it's never too late to correct the faults. Any veteran knows that the focus of any tester should always be the manual testing process, and this new information, when properly presented to a tester, can dramaticallyenhance the process of a manual penetration test:
· Optimization of the manual testing process - allow the tester to make better decisions, faster and test entry points that are more likely to be vulnerable first.
· Gaining Intel - enable the tester to understand how a certain page / entry pointbehaves under various conditions, by viewing a representation of the server-side source code, memory and cross-entry-point processes.
· Locate complex vulnerabilities - locate leads for vulnerabilities that require access tomultiple entry points, while overriding session and database values, with various perquisites and in extreme scenarios. Vulnerabilities that cannot be detected by automated tools, and are hard to locate even in manual assessments.
· Think about it… viewing the server-side source code of any component… criteria or not, it's simply awesome.
In addition, if the information can be delivered in a standard format to a black box web application scanner, it can enhance the coverage of the tool to include potential events and behaviors that only occur under extreme or rare conditions.
And what enables us to gather this information using nothing but black box techniques?
Well, I can only define it as... umm... breadcrumbs. Many tiny, seemingly useless pieces of information.
So if having the ability to gain Insight into the server side, reducing the time necessary to perform many types of tests and being able to locate vulnerabilities that nobody else can detect withoutsheer luck is of any interest to you, hang on a bit.
And just to make sure you're not losing track, here's one way to present it:
Activating Diviner's Clairvoyance feature - viewing a representation of the server side code
Viewing the Dynamic Server Memory & Processes Map Generated by Diviner
The process of manual penetration testing is a process of trial and error, which is composed of event-triggering attempts, behavior analysis and deduction;
Through a process of trial and error, the tester learns how a certain application entry point responds to specific input, access patterns and extreme conditions, locates behaviors that might be caused by potential vulnerabilities, and verifies (or rules out) the existence of these vulnerabilities through exploits, comparisons, etc.
Since there are dozens of potential generic application-level attacks (read the lists in OWASP,WASC and CWE if this number sounds exaggerated), excluding the use of scanners and fuzzers and with the exception of very small applications, this process can only be manually performed on part of the tested application entry points, and relies heavily on experience, intuition, methodology and sometimes luck.
The point I am trying to make is this - currently, there is an inefficient use of time in the process of manual penetration testing.
Don't jump to conclusions or take it personally... let me explain my intention:
Even though efficient information gathering enables the tester to narrow the list of tests that should be performed on each application, entry point, page or parameter - it still includes a lot of tests to perform, often more than the tester can do in the time allocated to the test.
Furthermore, since the most of the global information gathering processes rely on information disclosure, passive information gathering and fingerprinting, the tester needs to manually gather information on specific targets prior to testing them, or perform the test "blindly", while relying on other incentives.
Take SQL injection for example, one of the most common tests that penetration testers attempt to perform. In order to truly be certain that a certain location is (or isn't) vulnerable, the tester needs to receive different kinds of feedback; Sometimes a visible or hidden error make the task simple (blablabla.SQLException), Sometimes the tester needs to dig deeper and detect content differentiation, or compare responses to inputs that contain arithmetic or mathematical operations (id=4-2 vs id=5-3). When the tested entry point does not provide any feedback, he might be required to use payloads that are designed to delay the execution of SQL statements, and if an exposure with a similar obscure behavior affects an offline process or an indirectly affected backend server, he/she might even need to inject payloads that execute an exploit that alters content (risky) or sends a notification to external entities (mail, ping, etc).
Assuming the assessment method is a black box assessment, since there are various types of databases and syntax injection contexts, the tester will need to use a lot of payloads to truly verify the issue - in each field, and in each location.
Scanners attempt to tackle this issue by performing various tests on a wide range of targets, but conclude themselves whether or not the location is vulnerable, and currently, are far from performing these tests in a sufficient amount of extreme or complex scenarios.
Fuzzers on the other hand can store the different responses and behaviors of multiple entry points, but don't provide out-of-the-box support for complex processes or complex analysis methods, are usually not application-aware, and present the information in a way that is hard to digest.
The problem, however, could be handled using another method:
Divination attacks, a crossbreed between automated testing and human deduction, provide an alternate (or complementary) route:
Consider the methods required to detect the following complex vulnerability:
"SQL injection vulnerability, in which the *attack payload* is injected into a server variable in the *registration phase*, stored in the *database*, but only affects the application in the *event of writing an exception into a database log* (the vulnerable code segment), which only occurs in a module that generates the *monthly report* for a user, which requires *authentication*, while the log triggering exception requires the user to *directly access* the last phase of a multiphase report generation process while skipping the rest of the phases in the flow (forceful browsing)."
In other words, a vulnerability that affects the application indirectly, and only when certain extreme scenarios occur.
Although talented (or lucky) lucky testers might be able to detect it in a limited scope, it's unlikely that it will be detected by a black box automated vulnerability scanner, passive security scanner, or any other black-box tool… that is unless a certain process will make it possible…
Divination Attacks
When using the general term "Divination", this article refers to the following interpretation:
"Divination is the attempt to gain insight into a question or situation by way of an occultic standardized process or ritual. Used in various forms for thousands of years, diviners ascertain their interpretations of how a querent should proceed by reading signs, events, or omens." - Wikipedia's Definition for Divination.
Divination Attacks
When using the general term "Divination", this article refers to the following interpretation:
"Divination is the attempt to gain insight into a question or situation by way of an occultic standardized process or ritual. Used in various forms for thousands of years, diviners ascertain their interpretations of how a querent should proceed by reading signs, events, or omens." - Wikipedia's Definition for Divination.
For those of you that read this section first, and for those that got confused from the introduction, please, let me clarify: I am not proposing to hire the practitioners of witchcraft to participate in penetration tests.
I am however, proposing the following solution to the time management problem:
Inspect the direct and indirect effect of each parameter, on each page, with every possible sequence and under every possible condition, before deciding which attack to perform, and where.
Since obtaining this information manually is not probable, the process needs to be, at least in some aspects, automated.
And how can we obtain this information using an automated process?
Execute Scenarios -> Isolate Behaviors -> Perform Verifications -> Interpret -> GUI
Assume that interception proxy contains the following requests in its request history:
I am however, proposing the following solution to the time management problem:
Inspect the direct and indirect effect of each parameter, on each page, with every possible sequence and under every possible condition, before deciding which attack to perform, and where.
Since obtaining this information manually is not probable, the process needs to be, at least in some aspects, automated.
And how can we obtain this information using an automated process?
Execute Scenarios -> Isolate Behaviors -> Perform Verifications -> Interpret -> GUI
Assume that interception proxy contains the following requests in its request history:
The process must be repeated for the next "exit point", while sending another value (identical or otherwise) to the target parameter, prior to accessing the "exit point".
The result of this analysis might change due to various factors, such as:
Authentication - Authenticate before accessing the entry point, before accessing the "exit point" (a.k.a target), or not at all.
Multiple Sessions - When an entry point responds by replacing the session identifier, the scenario could continue using the old session identifier (assuming it was not invalidated), or using the new one.
History Requirements – Certain entry points might require the execution of previous entry points using a shared session identifier. For example, testing a parameter sent to the fourth phase of a multiphase process might require access to previous entry points using the same session identifier, with, or without authentication.
Input Type - The target "exit point" and "entry point" might respond differently to other types of input (e.g. input with random values, valid values, invalid syntax characters, etc).
Required Tokens – Certain behaviors might only occur when a required token is sent to the entry point (or not sent to the entry point) – for example, the existence of a timestamp or anti-CSRF token might affect each entry point in different ways.
Invalid Access – accessing pages without meeting their "requirements" might still generate a "beneficial" behavior – for example, accessing a page without a valid anti-CSRF token might trigger a response that reuses a server variable that can be affected, and thus, expose the entry point to attacks.
So in order to truly analyze the effect of the parameter on the various entry points of the application, we need to try everything (or at the very least – try a lot of scenarios), and we need to do it to as many input parameters as possible, to as many entry/exit points as possible, and in various scenarios.
Furthermore, the behavior itself might vary according to the scenario, input and in-page logic: it can be input reflection, exception, a certain valid response, time delay, content differentiation or anything else; the behaviors that we are interested in are behaviors that can be traced back to a certain process, memory allocation, potential issue or a specific line of code.
The information gathered in such a process will be composed of a lot of behaviors, which vary per page, per input, and per scenario.
These "behaviors" can then be presented to the tester in a simple, visual form, which will enable him to decide which behaviors he should inspect manually.
Don't get me wrong - I am not suggesting that we limit the inspection only to the information presented by such a process - I'm merely stating that it is wise to focus on this information first, and verify the various leads it provides before using the hardcore manual approach. After using this approach for some time, I can clearly state the following:
The information provided by the process, when used by a tester, can transformeven a very complex vulnerability into a low hanging fruit.
And that's not all. The collection of behaviors can also be "converted" into other useful forms, such as the ones presented in the following sections.
Source Code Divination
Source code divination is a new concept and approach (can also be referred to as source code fingerprinting).Authentication - Authenticate before accessing the entry point, before accessing the "exit point" (a.k.a target), or not at all.
Multiple Sessions - When an entry point responds by replacing the session identifier, the scenario could continue using the old session identifier (assuming it was not invalidated), or using the new one.
History Requirements – Certain entry points might require the execution of previous entry points using a shared session identifier. For example, testing a parameter sent to the fourth phase of a multiphase process might require access to previous entry points using the same session identifier, with, or without authentication.
Input Type - The target "exit point" and "entry point" might respond differently to other types of input (e.g. input with random values, valid values, invalid syntax characters, etc).
Required Tokens – Certain behaviors might only occur when a required token is sent to the entry point (or not sent to the entry point) – for example, the existence of a timestamp or anti-CSRF token might affect each entry point in different ways.
Invalid Access – accessing pages without meeting their "requirements" might still generate a "beneficial" behavior – for example, accessing a page without a valid anti-CSRF token might trigger a response that reuses a server variable that can be affected, and thus, expose the entry point to attacks.
So in order to truly analyze the effect of the parameter on the various entry points of the application, we need to try everything (or at the very least – try a lot of scenarios), and we need to do it to as many input parameters as possible, to as many entry/exit points as possible, and in various scenarios.
Furthermore, the behavior itself might vary according to the scenario, input and in-page logic: it can be input reflection, exception, a certain valid response, time delay, content differentiation or anything else; the behaviors that we are interested in are behaviors that can be traced back to a certain process, memory allocation, potential issue or a specific line of code.
The information gathered in such a process will be composed of a lot of behaviors, which vary per page, per input, and per scenario.
These "behaviors" can then be presented to the tester in a simple, visual form, which will enable him to decide which behaviors he should inspect manually.
Don't get me wrong - I am not suggesting that we limit the inspection only to the information presented by such a process - I'm merely stating that it is wise to focus on this information first, and verify the various leads it provides before using the hardcore manual approach. After using this approach for some time, I can clearly state the following:
The information provided by the process, when used by a tester, can transformeven a very complex vulnerability into a low hanging fruit.
And that's not all. The collection of behaviors can also be "converted" into other useful forms, such as the ones presented in the following sections.
Source Code Divination
Think about it - we use fingerprinting techniques to identify web servers, content management systems, operating systems, web application firewalls, and more.
Why not use the same approach to identify specific lines of code? Why not use it to detect all the lines of code, or at the very least, a large portion of the server code?
Nearly all of us classify source code disclosure, or attacks that can obtain the server source code as severe exposures (at least to some extent), and claim in the reports that we provide to customers that attackers can harness this information to enhance their attacks, learn about the system's structure and identify potential flaws in it.
If a large portion of the application's source code could be obtained using accurate "fingerprinting", wouldn't that lead to the same result?
In order to explain how this information can be obtained, let's use an example:
Connection pool exhaustion (or consumption) is one of the many forms of application denial of service attacks. It occurs when an attacker intentionally accesses an entry point (page/web service, etc) that requires a database* connection pool, using multiple threads – more threads the maximum amount of connections in the pool. The attack will delay the responses from entry points that rely on the pool, but won't affect entry points that don't use it (assuming the amount of threads don't affect other resources).
Although this behavior is an exposure in its own right, it also leads to the following conclusion:
It is highly likely that somewhere in the entry point's code, a connection is obtained from a connection pool, and since in many cases, a connection pool is a mechanism used to interact with databases, it's highly likely that the source code is similar to the following (jsp sample)
:try {
Connection conn = DriverManager.getConnection(…);
…
} catch (…) {…}
Of course – this connection pool might serve a different type of resource, but using additional verifications we might be able to increase the level of certainty – for example, identifying erroneous databases responses in the same entry point, or even detecting certain exposures in other application entry points.
The same approach can be used to convert other behaviors to the lines of code that might have caused them, and since the previous process gathered a lot of behaviors – these can be converted into a fair amount of code - pseudo code that can be presented using any specific syntax, and enable the tester to understand how a certain page behaves – prior to testing that page.
For example, input sent from one page (the "source" page), but reflected in another (the "target" page) is probably shared through a session variable, file or database field. The origin can be isolated by accessing the target page using a different session identifier, but using the same identical process used to access it before (login, history, etc) - with the exception of the source page;
If the reflected input is not present in the target page, the probability for the existence of the following lines of code in the source page and target page increases:
Source Page:
String input1 = request.getParameter("input1");
session.setAttribute("sessionValue1", input1 );
Target Page:
out.println(session.getAttribute("sessionValue1"));
If however, the reflected input would have been present at the verification scenario, than the source code matching the pattern will probably include database access, file access or static server variables – and specific aspects of these behaviors can be isolated in turn (insert statements are more likely to exist in pages that rapidly increase in size, update statements in pages with relatively static size and persistent changes, etc).
At the end of the processes, after performing additional verifications and tests, the options with the highest probability will be selected and presented to the user.
And how will this code be sorted? Which lines will appear first?
Although the sorting problem has many solutions, one of the main solutions is probably "delay-of-service" attacks (yes, I said delay, not deny).
Presented in the research "Temporal Session Race Conditions", these attacks were originally meant to delay the execution of specific lines of code, in order to extend the lifespan of temporary session variables – but these attacks can also be used to sort some of the code – by inspecting if exceptions or conditional behaviors occur instead of the delay, before the delay, after the delay or not at all.
For example, performing a connection pool exhaustion attack on a page while simultaneously sending an error generating value to the same vulnerable page will provide a potentially important piece of information – which code is executed first: the code that attempts to obtain a connection from the pool, or the code that is prone to the exception.
Note - Although this method isn't exactly "safe", it will probably enhance the results more than other methods for sorting divined lines of code.
Like fingerprinting, this information might not be 100% accurate (although it can be VERY accurate, if the processes is performed properly and thoroughly), but can still be very beneficial for the purpose of the test – just like other forms of fingerprinting.
I won't expand the subject of source code divination in this post (I do have plans to discuss it further in separate posts), but it's already implemented in the diviner extension that will be discussed in the following sections.
Connection conn = DriverManager.getConnection(…);
…
} catch (…) {…}
Of course – this connection pool might serve a different type of resource, but using additional verifications we might be able to increase the level of certainty – for example, identifying erroneous databases responses in the same entry point, or even detecting certain exposures in other application entry points.
The same approach can be used to convert other behaviors to the lines of code that might have caused them, and since the previous process gathered a lot of behaviors – these can be converted into a fair amount of code - pseudo code that can be presented using any specific syntax, and enable the tester to understand how a certain page behaves – prior to testing that page.
For example, input sent from one page (the "source" page), but reflected in another (the "target" page) is probably shared through a session variable, file or database field. The origin can be isolated by accessing the target page using a different session identifier, but using the same identical process used to access it before (login, history, etc) - with the exception of the source page;
If the reflected input is not present in the target page, the probability for the existence of the following lines of code in the source page and target page increases:
Source Page:
session.setAttribute("sessionValue1", input1 );
Target Page:
If however, the reflected input would have been present at the verification scenario, than the source code matching the pattern will probably include database access, file access or static server variables – and specific aspects of these behaviors can be isolated in turn (insert statements are more likely to exist in pages that rapidly increase in size, update statements in pages with relatively static size and persistent changes, etc).
At the end of the processes, after performing additional verifications and tests, the options with the highest probability will be selected and presented to the user.
And how will this code be sorted? Which lines will appear first?
Although the sorting problem has many solutions, one of the main solutions is probably "delay-of-service" attacks (yes, I said delay, not deny).
Presented in the research "Temporal Session Race Conditions", these attacks were originally meant to delay the execution of specific lines of code, in order to extend the lifespan of temporary session variables – but these attacks can also be used to sort some of the code – by inspecting if exceptions or conditional behaviors occur instead of the delay, before the delay, after the delay or not at all.
For example, performing a connection pool exhaustion attack on a page while simultaneously sending an error generating value to the same vulnerable page will provide a potentially important piece of information – which code is executed first: the code that attempts to obtain a connection from the pool, or the code that is prone to the exception.
Note - Although this method isn't exactly "safe", it will probably enhance the results more than other methods for sorting divined lines of code.
Like fingerprinting, this information might not be 100% accurate (although it can be VERY accurate, if the processes is performed properly and thoroughly), but can still be very beneficial for the purpose of the test – just like other forms of fingerprinting.
I won't expand the subject of source code divination in this post (I do have plans to discuss it further in separate posts), but it's already implemented in the diviner extension that will be discussed in the following sections.
In the previous process, we have discussed how an identified behavior (such as an exception or input reflection) can be classified as persistent or temporary – by reproducing the scenario that caused it using a different session identifier, identical process, and without accessing the "entry point" (source page). This process, alongside additional verifications allowed us to conclude whether a behavior is persistent, temporary or something else.
Although not all the behaviors rely on specific variables that are stored in the server side, some do, and from these behaviors we can conclude how and where does the server stores some of the content.
By crossing the information obtained from interesting scenarios that were discovered in the process, we can even locate multiple entry points that affect the same database tables, fields, session variables and static variables, and thus, construct a general structure of database tables and session attributes.
The Diviner Extension
During the last year, I collaborated with a number of individuals (especially with @Secure_ET, various colleagues and the OWASP ZAP project) so that these ideas will not remain a theory… and after numerous late night brainstorming sessions, various incarnations and a long development period – we have an initial version that works (beta phase).
The diviner platform – an active information gathering platform that implements many of the previously described concepts, is implemented as a ZAP proxy extension, and can be downloaded from the following address:
http://code.google.com/p.driver
It can already illustrate server side behaviors and processes, contains features such as the task list/advisor which provide invaluable leads to potential exposures, present a partial map of the server side memory, and present a partial representation of the server side code.
The extension is deployed using a windows installer (or in binary format for other operating systems), and requires java 1.7.x and ZAP 1.4.0.1 in order to run properly.
Furthermore, since it attempts to identify behaviors that result from valid & invalid scenarios, and can't guess what is valid on its own, it must be used after a short manual crawling process that covers the important application sections with valid values.
It was tested mostly on small scale applications (100+- parameters, +-50) – including real-life applications, and although it will probably work on larger applications (it's not stuck in the database analysis process – be patient) – due to various optimizations (and sacrifices) we didn't yet make – it's recommended not to exceed that size.
We can currently identify 20+- different lines of code, but have plans to implement tests that identify other lines of code, some with high probability, and some with absolute certainty.
We didn't yet implement features that sort the lines of code (and thus, currently rely on default positioning), but plan on implementing them in the future (with restrictions that will prevent their use for actual denial/delay of service attacks).
We have many additional experimental features that aren't mature enough, but are already working on refining them for the future versions.
We don't perform any form of automated vulnerability scanning, but plan on exporting the interesting leads to a format that can be used by external scanners to detect exposures in these abnormal scenarios.
Bottom line - It's not perfect yet, but it's already very useful, and can already help testers locate exposures that can't be located using other means, and make better decisions - quicker.
Download Link Demonstration Video :
Source Code Divination
Persistent SQL Injection
Persistent XSS Detection
No comments:
Post a Comment