The idea of providing databases as a service to customers and companies can be a very foreign idea to the database administrator.  The DBA’s job is to carefully control and dole out resources.  The concept that “in the cloud” the user can just “order up” a database on demand is a difficult one.

How does it work?

Infrastructure

First, the a cloud-based Paas Infrastructure zone must be created.    Without this, the plug-in for the “database as a service” will function.  Once this is setup, then the “Database for a Service” can be completed.

The DBA must create a couple users for the distinct purpose to managing the Paas Infrastructure and the “Database as a Service”.  For the Paas Infrastructure user, once the user is create, you must grant the EM_CLOUD_ADMINISTRATOR role to it.  The “Database as a Service” user must be created and granted the EM_SSA_ADMINISTRATOR  role for it to manage allocation of resources, etc. for users that will choose the “Database as a Service” option.  Any users that are going to utilize the database as a service feature will require the EM_SSA_USER role and the role(s) that will be granted to the infrastructure zones that are created in the upcoming steps.

This is a middleware feature, so keep in mind that this is accessed from the middleware menu drop-downs. You can deploy the infrastructure server pools and zones to control what resources are allowed to be allocated to the each request.

I would recommend setting up roles that you will wish to allocate access to use the database as a service privilege.  You may have one or multiple ones depending on hosts that will be assigned to different cloud groups, development vs. testing, etc., but grant these roles now, which will ease the set up once you set up your zones.

If no Paas Infrastructure exists, it will take you into the setup wizard. You will choose your VM host(s) that will be part of the Paas infrastructure zone, along with any roles you want to grant to the first zone.  Each zone should be treated like you would separate development, test and production or business area or client.  You may have multi-tiered zones which separate development for one department to one VM host, while having a second zone to testing for another department on another VM host.

You will decide how much total memory, CPU and desk space that will be allocated to the zone.  The wizard will ask you to review the choices on the zone before you finalize the zone setup.  Once you have set up each of these infrastructure zones, then the “Database as a Service” an be setup.

Database as a Service

To perform the next step in the setup, you must be logged in as the user you created and granted the EM_SSA_ADMINISTRATOR role to.  This user will then access the wizard by clicking on Setup –> Cloud –> Database.

The first tab has you allocate Database pools that consist of servers/hosts that will make up each database pool.  This feature is dependent on Oracle database software installed  on each of the servers that will be part of each database pool.  Remember that all servers that reside in one pool must have the same version of OS, database software but the Oracle home can exist in several locations and chosen as part of the steps when requesting a database later on.  Named credentials are required for the OS level privileges for the database as a service feature to function.  If you do not have them already set up, ensure to configure them and save them off for future use.  Review your setup and save, but keep in mind that any changes that are required can be performed via the “Edit” option fro the main database pool screen.

The next screen is the “Request Settings” tab.  This screen can automate all settings for requests, including when to delete requests or when to archive.

The next tab, “Quotas” is of significant importance, as it states who assigned access to what Database pools, have what resource allocation they can utilize in their requests.  These are assigned at the role level that we set up in the Paas Infrastructure configuration and now we can allocate amount of memory, CPU, number of schemas and number of databases that each role has privileges to create.

Profiles and Templates tab takes the DBA through the steps to set up sample schema designs and databases for specific database or application use. There are a number of ways that this can be setup, including RMAN backups, (which uses an RMAN duplicate to perform the task…) full database template, new or existing DBCA template or schema template.   If you already have DBCA templates that are saved off for use on the host these can be used in this area to be used for provisioning of the “Database as a Service” requests.  You an save off these templates in the software library for future use and access by the requester by location and database version.  Once submitted, the profile is created and available for use.

By now accessing the Database Cloud Self Service Setup wizard, We will create the service template, the last step in the process.  No plug-in is required, just choose Create –> database/schema.  For this example, we are going to stick to “database”.   The previous template we created was for a database template, so it should be selected and will show in the list, along with any others that have been created and saved.  If it’s a single instance template, then the service template should be set up for single instance vs. RAC, which is also an option.  You can then choose what database prefix you desire, which will control some of the crazy database naming that *could* and very may likely occur, (yes, the requester will choose the database name prefixed by what you choose here….) along with domain name to control the global name settings.  Listener port settings are next.  I didn’t see any way to use multiple port options per zone, but for RAC, a scan listener setup was available.

The next window in the setup will look very familiar to any DBA.  It looks very similar to the setup for databases in the DBCA, (Oracle’s Database Configuration Assistant utility).  This allows the DBA an easy setup via ASM, but will require a defined directory path for datafiles.  This can be difficult for Windows VM users if all drives are not the same for the host pool, (i.e. one server has the G:\ drive available vs. another that has the F:\ drive.)  When Server admins are designing the cloud infrastructure to be used for Oracle and Windows is the OS platform, ensure they understand that having mirror image servers is important.

Security settings in this setup are going to make most DBAs a bit tense.  Yes, all of the databases in this database pool will be have the same password for the SYS, SYSTEM and DBSNMP.  The max configuration complexity is to set it different for each of the three users so that they have individual passwords, but all databases created through this template will have the SYS set the same for whatever you type in for the SYS password field, SYSTEM, etc.

Setting up the flash recovery area will be dependent upon the VM servers assigned to the database pool to be mirror images to each other just as the datafile location will be unless ASM is used.  Setting up a default value for this may be hit or miss, as it may be difficult to judge how each database is used in the end, even if the templates are carefully designed and monitoring for FRA usage through the EM12c should be set up with significant metric thresholds that support this vulnerability.

The service template must be allocated to an Infrastructure zone, as was created in our initial steps.  Once that is done, you will be asked to add your roles to the service template created earlier on, that will then offer those users rights to use this template.

As with any cloud service, it wouldn’t be an EM12c cloud feature without charge back.  Maintenance features for charge back can be accessed from the Enterprise –> Chargeback menu in the dropdown, but for this post, we assume that charge back plans have already been set up, so we will simply continue with the Chargeback tab.  For anyone who is unsure how to set up charge back, I highly recommend ”Enterprise Manager 12c Cloud Control Metering and Chargeback“ to get a feel for this complex task.

The charge back plans are then assigned to the actual Paas Infrastructure zones.  Doing this will ensure that anyone creating databases in these zones are charged correctly for usage of these cloud resources.

Using the Self Service Portal

Within the Self Service Portal, the user can go to the “My Databases” tab click on “Request” to start the process.  The request name will be automatically filled with the user name and the date, along with choices made by the privileges allocated to the individual user.  Single instance/RAC, amount of memory/CPU required, time to deploy the database, (immediate or scheduled) and then create a user for instant access for the administration of the database.  This is an EM job submission, so the job does exist in the EM job tables.  Provisioning will show the progress of the database creation and if any errors or issues arise.  Once completed, we can then re-enter the Self Service Portal and see the database listed in “My Databases”.

Managing the Database

By double-clicking on a database in the “My Databases” list, access is given to the main management page.  The user is able to start/stop the database, view connection information, if any administration work is required, (including backups if they are not already configured) and basic performance information.

Pros and Cons

To have the ability to create a database for development, testing or QA without having to allocate over-stressed DBA resources is a great option.

• DBAs will most likely limit the use to production VM access to the feature to leads in departments or DBA’s for customers of their cloud services.
• Non-DBAs need to understand that backups of databases are not automatically set up when the database is created, so creating scripts to setup backups may be beneficial that can run at the end as part of the service template is recommended.
• Templates can include other post-database creation scripts to automate even more tasks that were once on the DBAs list of demands.
• The security setup is limited.  If SYS/SYSTEM is discovered in one database involved in a template used, it could be used to access other databases.
• Clear understanding of how charge backs work must exist if they’re used.

DBAs should take the time to dig into the “Database as a Feature” and think about the benefits to future in how it will free up their time to take on more fascinating challenges.

No fear of the future, it’s already here….

the user who has been granted access to this feature, will be able to request a database, just like any other automated process that is available via the console.  To perform the task is not any more complicated than requesting a report via an application on the web.

As many know, I joke about being a strange, ADHD kid.  I didn’t really fit in with the other kids and was really, really small for my age.  I was bullied and very much a loner.  Growing up, my closest friend was my middle sister, Kristi who is 2 1/2 years younger.  She was the golden child-  blond hair, blue eyes and very, very sweet.  She was so easy-going that she was often frustrated by other kids, so my easy-going, but, hyper personality was preferable to them and we were happy to hang out together.

We grew up in a very rural area of upper Michigan and spent our days playing make believe in the fields and forests around our house on Lake Michigan.  When we did move to higher population areas, first in Oregon and then in Virginia, we still hung out with each other than our school mates, even though we attended different schools.

I was the maid of honor at Kristi’s wedding to her husband, Jim and we’d visit when I’d come out to Oregon to visit about once a year.  We’d pick up like we’d never been apart, which is common for sisters.  I was happy to see that she and our youngest sister, Kimberly, became very close as they had grown up and they have continued with that relationship.  It’s hard not to like Kristi, she is simply “likeable”.

She turned 43 this year and as she’s always been a bit of a health-nut, appeared to be in the prime of her life.  She had no clear “cancer indicators” when she started having symptoms in January and it wasn’t until February that the doctor’s discovered she had what appeared to be colon cancer.  Within weeks, scans came back showing she not only had colon cancer, it had spread to her liver.

Both my parents have had their challenges with cancer.  My Mother had a battle with uterine cancer and my father, a version of leukemia, but both bounced back rather quickly with treatment.  We respected Kristi’s desire to be private and expected her to make a full recovery.

This hasn’t happened.  She is now, just a couple months later, stage 4 colon cancer.  No treatment has hindered the progress of the cancer’s spread through her body and she is in her final battle.  The ravages of the disease are not just physical, they have brought a horrible financial burden as her family has struggled to save her.

The feeling of helplessness is difficult.  I, as many in my field can understand the need to “fix things” or at least make things better and there is so little I can do.  My sister’s friend’s have put together a contribution site to assist in their financial struggle to help with the cost of her treatments.  It makes the pain a little less knowing I can help ease the financial burden even if I can’t do anything to help save my sister.

If you would like to help Kristi’s and our family, click here.

Many of you know that I’m the mother of three, very bright, “equally ADHD as their mother” children.  As I’ve been tasked with a number of interesting challenges with both RMOUG’s and ODTUG’s WIT, (Women in Technology) events, I’ve started to investigate how, no matter what we do for women today, unless we start focusing on the women of tomorrow, we are lost.

I spent some time the last couple weeks investigating the offerings by my local school districts for technical education and am alarmed for both genders.  There are “Technology Plan Templates” that detail out the dismal specifications of what hardware will be provided in the classroom, but no mention of what to attain with it.  No one lists out what educational goals, what students hoped to achieve with the said technical hardware.

Now what were these marvel in educational opportunities offered in the way of software?

• Windows XP.
• Tutorials on how to use a USB flash drive.
• How to “Navigate the network”

The labs were beginning classes in Microsoft Office and how to use email.  What kid, by the time they are in mid-elementary, lack knowledge in9 basic tasks in MS Office??

I looked on the clubs link, as I remember computer and programming were often found in clubs, along with classes, in my high school.  This was what is now offered:

Sigh….

I assumed the technical classes had to be missing from the website, (I know, seemed like an oxy-moron to me, too, but I was willing to give the benefit of the doubt…. )  so I called the school.  I was politely informed that all classes for any student that are interested in technical educatiin are offered at Bollman Technical Education Center.

I remember this school, as it was what was recommended to my oldest when he was having challenges that were administration in nature, but had been classified by the school as “behavioral”.  If this doesn’t concern you right off, we’ll discuss it here in a minute, but first I wanted to make sure that there hadn’t been a “revamp” of the school.

No, Bollman is still considered a school that “benefits the student who is more productive when doing than learning in a more traditional academic setting.”

I was less than star-struckwhen I looked at their technical career programs:

Auto technology?  Do we really even have to discuss this one?

I was more pleased by the clubs, but the ones that are heading it up are often the same teachers that are failing the students in their primary schools!  I noted the names of three teachers that aren’t even technical in the three clubs that are mentioned in the image below!

One of the teachers listed as one of the sponsors teaches at my children’s high school and we’ve discussed her “ongoing challenges grasping today’s technology”.  Having some of these folks heading up tech clubs seems no better than the BBC show, “The IT Crowd” character, non-technical Jen Barber’s hilarious attempt at heading up IT, which is one of the consistent comedic story lines of the show.  This is what we have directing tomorrow’s technical minds?

Now we’re going to get into the down and dirty.  This school is an “alternative” school.  This is the school that they send the students that are not succeeding in standard, public schools.  What does this say about the future of technology in general?  What does this say for the number of folks in the industry that have a story similar to mine that started out in a VERY different field than the one they are highly successful in because they were unaware that the opportunity was even out there for them?

So, here’s where I’m at-  if the schools are going to continue to fail both genders in the technical arena, maybe this is a better alternative:  http://www.technologyreview.com/news/506466/given-tablets-but-no-teachers-ethiopian-children-teach-themselves/

How many performance specialists have experienced projects where reporting was horrid in the beginning, maybe didn’t even complete?  The focus of the project may have sat with data loads or due to the data architect’s design specialty, etc., the reporting aspect was just the last thing on everyone’s mind until it was too late. I come into this conversation with companies all the time and it is a difficult one, as there are so many database myths.

“Can’t you just do something to make the database go faster?  I don’t know, isn’t there a parameter that can be updated, maybe some partitioning or compression that will fix our problem?”

I feel there pain, I do understand…

When a project to design a warehouse or mart is undertaken, the groups involved may be quite distracted by following “best practice”, i.e. “my fact tables must be just so and the dim table design must follow the same format as I used at company XYZ that was so successful….”  You may find that the folks doing the actual design may not know as much about marts and warehouses and are learning as they go along.  You may have someone who is unable to move away from an OLTP design in the mart, “everything must be relational and I must follow normal form, so let me add primary keys, foreign keys, constraints everywhere!”  (Hey, it happens, I’ve seen it…:))

Never the less, as a development group learns, there is no place I’d rather be.  No project is a lost cause and working with the group to create proof of concepts, showing them to correct way to design a warehouse or mart, along with star schemas, the power of bitmap indexes, etc. is a valuable time for any database administrator to spend.  Ensuring you know what is causing the pain for an ETL load process, i.e. where time is consumed is essential, so that you are able to provide as many “wins” when making enhancements is pertinent, so you can get onto the reporting aspect of the project.

So why do we need to keep reporting on the horizon at all times?  Never lose sight of this important feature of the warehouse or mart as this will most likely be the user interface to the environment.  This is how the business will judge if its successful or not.  If they can’t view the data in an acceptable amount of time, it doesn’t matter how quickly it loads at night or how beautiful your star schema is, as far as they are concerned, it was all a waste of money and time.

With that said, where do I see the most common challenges for reporting?

Storing data in a very different format than the way reporting requires it to be presented in. 

I’m not talking about how a star schema is designed here.  I’m talking about the data having to be constantly cased, then summed then sorted, hashed, sorted again, so on and so forth, to present the data the way the customer requires vs. having materialized views, reporting and rollup tables that can lessen the performance hit on the database.

Why is it a performance hit?  I have another draft in my blog posts that I need to get out, but it’s a perfect example.  The business needs to see the data presented consistently in a way that is no way near the way its stored.  As the reporting queries perform well with simple hashes, bypassing nested loops, etc., there is no way that the amount of case statements, to sum, to sort and sort again, to fit within the allocation of PGA per process for it, even in parallel.  This results in huge uses of temp tablespace.  That means the process is now doing it’s work on disk and disk is just plain slow.

Why am I a temp tablespace nazi?  This above is why. 

I can move to temp tablespace groups and I can put my temp on SSD, but really, the best optimization enhancement will come from an materialized view or rollup table that will limit the amount of work that has to be done in PGA.   Do not say, “Well, can’t you just raise the PGA?”  These are on servers that the amount of PGA required is larger than the memory on the box and the temp tablespace is often the largest tablespace in the database, (remember, I work in VLDB’s and for someone to brag that they used 2TB of temp is not uncommon… )

No hardware will offer the same kind of performance gain as a small design change along with a few tweaks to some reporting SQL to do less in a report to produce a report.  No hardware will offer as long-term performance enhancements as the latter-  plain and simple.

I wish I did have a silver bullet to make the database “go faster”, but that’s not how it works if you really want the problem fixed vs. having someone offer you some duct tape.  Duct tape will last you a month, maybe a year if you are lucky, but the problem will continue to return until we fix it right.

OK, enough ranting for my first day as a ACE Director…

I won’t apologize to Oracle for saying that the new Incident Manager has some quirks to it, but it does, especially when it comes to managing incidents, events and how to clear out both.  I’m going to attempt in this post to explain how to check to see if you have alerts that can be cleared with an EMCLI command and then also take you through clearing out from the GUI, which is available in the Release 2 of EM12c.

If you want to know what kinds of stateless alerts that can be cleared manually, then once logged into the EMCLI, (Enterprise Manager Command Line Interface) you can do the following:

./emcli login -username=<login name>

./emcli get_metrics_for_stateless_alerts  -target_type=osm_instance

This command will return all metric types and alert types that can be cleared out for ASM alerts, (osm_instance).  You can do this for oracle_database or any of the other target_types.

As always, if you need help, EMCLI offers a good help option.  You can save off the file for reference and then gather more help info on specific verbs by doing the following:

./emcli help > emcli_verb.lst  <--creates the output file of the emcli verbs and descriptions.

./emcli help <verb>  <--gives you a descriptions, syntax and examples for a specific verb.

If you aren’t sure of Target Type names, you can query the database or you can do wild card searches through the EMCLI:

./emcli get_targets -targets=%database%

./emcli get_targets -targets=%osm%

./emcli get_targets -targets=%

This first option above returns a list of all targets that are database type targets.  The second returns all ASM targets and the third returns everything.  Some of the output from the third looks like the following:

ID      Status           Target Type           Target Name
0       Down             oracle_listener       LISTENER_SCAN2_exa1-cluster
1       Up               oracle_oms            oemdb02.orcl.com:48
1       Up               oracle_oms_console    oemdb02.orcl.com:48 
1       Up               oracle_oms_pbs        oemdb02.orcl.com:48
1       Up               oracle_database       orcl_db3.world_orcl2
1       Up               oracle_database       orcl_db4.world_orcl1
1       Up               oracle_exa_pdu        exa1-pdua.orcl.com
1       Up               oracle_exadata        exa1cel07.orcl.com
1       Up               osm_cluster           +ASM_exa1-cluster
1       Up               osm_cluster           +ASM_exa1-cluster
1       Up               osm_instance          +ASM1_exa1db01.orcl.com

Before attempting to run a clearstate on incidents, lets look at how to check for these.Note that we use two arguments here with the clear_stateless_alerts verb:

• Unacknowledged_only-  Only those alerts that have not been acknowledged, which helps distinguish from any that may be currently worked by someone.
• Preview-  I want to just check and see how many are available to be cleared.  The command will not clear anything this way, only tell me how many!

$ ./emcli clear_stateless_alerts -older_than=0 -target_type=oracle_database -target_name=<target name> -unacknowledged_only -ignore_notifications -preview

 The following alerts can be cleared:

Total Alerts
==============
1

Now, if you want to clear the one alert that is available to be cleared as stateless, you can:

$  ./emcli clear_stateless_alerts -older_than=0 -target_type=oracle_database -target_name=<target_name> 
1 alerts were cleared successfully.

Now if you have a huge quantity of alerts that are not available for clearing with this process, the EM12c console Incident interface offers the ability to mass clear alerts in release 2, (apologies to anyone still using BP1 and lower, this option will not work!)

From the EM12c console, there are a number of ways to access alerts, via the Incident Manager from the Enterprise drop down or from a database target summary page, but for this example, we will work from groups.

By clicking on All Targets, then Groups, we are brought to the groups summary page.  As we see, our Prod group has 932 open incidents identified with this group:

By clicking on the incidents link, (which is the number of incidents, so for this group, we would click on the “932″) we are then taken to the overall incidents summary for this group.

If you hover your cursor over the left hand side column, just left of the Severity column, you are able to hold your left mouse button and choose multiple alerts:

By dragging the mouse down, you can choose as many incidents/events as you wish.  Once you have chosen the amount you wish to remove, note that the “Rows Selected” will show in the left bottom corner, (for our example, we have 7.)  You can then click the “Clear” button, identified with the pencil eraser.

The confirmation pop-up screen will then ask you to confirm and click OK, but remember to change it to not send email notifications so as not to “spam” everyone.

The worst thing you can do is send ridiculous amounts of emails telling everyone you are removing each of these incidents.  Remember, we hate “white noise”…   Once you have confirmed this is what you want to do, click “OK”.  You can do this for as many or as little incidents to clear out the system as you would like at a time.

Remember-  Any incidents that are still an incident and need attention will recreate on the next metric collection for the target.

As many of you know, the WIT Session at RMOUG and subsequent planning for a WIT program has been very successful.  I just received my evaluations, as this was offered just like any technical session.  There was a wonderfully, honest comment that I hope the author will not mind me sharing anonymously and I hope my response will help her in return.

“to be honest, if I could [do] it all over again, I wouldn’t be a DBA. I have a degree in CIS, computers were the in thing when I was in college. I chose CIS for the money to support myself & my child. It was something new & exciting. Now it is stressful and being on call isn’t fun. If I could do it over again I would have done something in the medical field like my sister who is a doctor. A lot of my colleagues say the same thing about not doing IT field as well if they had to do it again. Honestly, I believe that is why you aren’t seeing young girls & boys getting into the field because they see what their parents do and how stressed they are, etc. in the IT world.”

I found this comment both sad, honest and interesting. Facebook’s COO, Sheryl Sandberg, who just wrote a fantastic book, “Lean In”  discusses women in the workforce, not just IT and one of the things she says is crucial is a solid support system for any woman, especially if she has a family. This includes her family, her company and her manager.

I also have a couple friends in different areas of the medical field, I find that their schedules can be as stressful as the worst IT one, especially nursing.  I wondered if her sibling’s job choice in companies vs. the attendees was more the reason behind satisfaction than the career.  In my experience and from what I’ve seen, a good company is worth twice the actual career choice.

I admit that I entered the DBA field because of the flexibility it offered me.  I first became a DBA when my middle of three children was just an infant. The kid’s father was raised in a household with a stay at home Mom.  I think its challenging for a father that lived with a Mom who’s job was to take care of the family, to then have a wife with split responsibilities and higher demands than what he may have experienced his father having.  Some men acclimate to the change fine, others do not and my ex-husband did have difficulty with this.  With a career in IT, I could take time off from the office and take the kids to doctor appointments or go to parent/teacher conferences.  I could work from home if one of the kids were sick that day.  The good pay and ability to telecommute for after-hours tasks worked well with my lifestyle and my children were comfortable with Mom being present, with a laptop as her constant companion.

As I mentioned that the kid’s father is my ex-husband, yes, I was a single parent for many of those years.  I have a good parenting relationship with my ex, so no doubt that really does help, but he has to admit that I’m the one that goes and gets them from school, the one that stays home when they are sick and the one that addresses all work hour requirements.  I would have found it difficult to locate another career such as database administration that offered me this type of flexible schedule, as well as the ability to telecommute full time, which I’ve now done for two of my positions.

In the last year, I’ve begun to travel to present at conferences, where before 2012, I only presented locally.  I now am presenting at about 4-6 remote conferences which requires me to travel and my ex-husband and his new wife care for the kids during that time.  The kids are now 12, 15 and 18 yrs old, so the care is minimal and they don’t complain too much about having their schedules interrupted for a couple days.  Outside of the few days I travel once every two months, I work from home.  I’m here every day when my kids get home and I’m able to pick them up from school or take them to appointments.  Pretty much whatever they need me for, I’m here.  I have two office areas in the home and even if I’m not home, I’m incredibly accessible via cell, email and chat.

Although I understand where the attendee to the session is coming from, I’m going to focus on where I think the real problem lies-  I’ve seen IT environments where they work people 12 hrs a day and require them onsite, refuse to comp time, etc.  This is a work culture issue and not a database administration career issue.  We see it in not just IT jobs, but in so many others as well.  IT often derives from poor management and poor work culture.

I would say to this wonderful woman, “Please, do not give up on this career!  Search out companies that support their staff’s lives, realizing the difference running a company hard and running a company smart!”

I’ve worked in these environments.  Interviewed where it sounded very wonderful and no hint of what was in store, but once you are in the doors, you are thinking, “My God, what have I gotten myself into?”  You are torn as your family has demands of your time and you have a boss that doesn’t understand why the family can’t come second, third or “can’t your Mother-n-law just take care of it?”  Sigh….

I made a pact with myself a while back.

• Only work for good bosses and for good companies.
• Work for companies that support you in your own goals and family life, not just for their priorities.  It’s should be a win-win with small compromises that pay forward.
• Companies that support their employees have more loyal employees and are more likely to succeed.  The employees work for the company because they want to, not because they have to.
• Realize your worth and work hard for what you want.
• Don’t listen to those that tell you that you can’t accomplish something or that something will never happen.
• Surround yourself with positive people and find mentors
• Yes, there is often oncall/after-hours work with a DBA career.  This is why I recommend any DBA’s first goal is to silence the pager and remove all “white noise”.
• Any manager/department that is a proponent of ridiculous tasks allocated to after hours work, extended hours where a resource is exhausted and not able to function well and/or “white noise” paging-  START LOOKING FOR A NEW JOB.

I don’t know if this will reach the woman from my WIT session, but for any woman in the industry that feels this way, I hope it helps them find their way to succeed.  I love my job, my company and my career.  I wouldn’t change my path and yes, I’ve experienced many bumps in the road, but to have the persistence to continue on the path that is right for me is the only way to success.

There’s an old Hebrew proverb-  “Fall down seven times, get up eight”

The eighth time rocks.

Exachk does a number of health checks.  One of them is to verify the connectivity, settings and such of the ethernet cables in the Exadata environment.  If one is found amiss, the following will be returned in the Exachk summary report:

To validate the Exachk findings, you are asked to run the following as root on the specific node that is reporting the issue:
for cable in `ls /sys/class/net | grep ^eth`; do  printf “$cable: “; cat /sys/class/net/$cable/carrier; done

This command, if run on one of the other nodes that were not showing up in Exachk without an issue, the following is returned:
eth0: 1
eth1: 1
eth2: 1
eth3: cat: /sys/class/net/eth3/carrier: Invalid argument
eth4: 1
eth5: 1

Running this on third node, where the report of the failure had occurred and the following is returned, clearly reporting the issue on eth1 and eth4 that requires a physical check of the connections reporting back with a value of “0″.

eth0: 1
eth1: 0
eth2: 1
eth3: cat: /sys/class/net/eth3/carrier: Invalid argument
eth4: 0
eth5: 1

“Invalid Argument” signifies that it is not in use, so disregard eth3.  The values of 0 are the issue.  This shows we have a problem.  Andy Colvin, (our very own Enkitec Exadata Ninja!) recommended doing the following to just check a bit further.

As root on the third node, issue the following: ethtool eth3

$ ethtool eth3
 Settings for eth3:
 Supported ports: [ TP ]
 Supported link modes:   10baseT/Half 10baseT/Full
 100baseT/Half 100baseT/Full
 1000baseT/Full
 Supports auto-negotiation: Yes
 Advertised link modes:  10baseT/Half 10baseT/Full
 100baseT/Half 100baseT/Full
 1000baseT/Full
 Advertised auto-negotiation: Yes
         Speed: Unknown!
         Duplex: Unknown! (255)
 Port: Twisted Pair
 PHYAD: 1
 Transceiver: internal
 Auto-negotiation: on
 Supports Wake-on: pumbg
 Wake-on: g
 Current message level: 0x00000003 (3)
   Link detected: no

Yep, the lights are on, but nobody’s home…

At this point, someone *physically* needs to check the hardware.  For the incident above, it was suspected that someone had yanked on the cable, damaging it and due to this it was replaced, ran through the above tests again to verify that all was well.

As I spend more and more time optimizing Exadata environments, I seem to be answering the same questions repeatedly when it comes to database design, parameter settings, etc.

Here’s a few high level notes to always remember-

• Always have baselines from previous environments.  Don’t go into performance expectations blind.  Have AWR and ASH data from the previous environment and mine AWR data for anything when others start “guessing”.
• Look at the big picture-  those that focus on one thing, often find that it’s a combination of issues that brew the perfect storm.
• What are your top 10 goals by going to Exadata?  They had better tell you it wasn’t just because someone thought “it was cool”.
• Expectations should be listed clearly, “we are expecting a XX% average improvement in our ETL loads and XX% average improvement in our reporting.  We expect to have a XX week post optimization exercise to address any performance issues after the migration and know that this may involve query rewrites.”
• Do the challenges in your database environment coincide with the features that Exadata offers?  If not, you could have a really sweet engineered system that offers your current design minimal improvements and what that leaves you with is future enhancements that should be geared to utilizing smart scans, storage indexes, HCC, etc.
• Just because a statement offloaded and returned the data quickly, doesn’t mean you have unlimited PGA to then sort and aggregate that data in a report.  Cell nodes can not help you here.

“My goals are not your goals” is an opportunity for any DBA who hears these words…

• Have you identified your stakeholders in the optimization exercise?  What enhancements are important to you, may not be what is important to them.
• Discuss what change control is required.  Ensure you know what is required to move anything to production.
• Openly listen to all involved and realize that all input is valuable and in the end, if the user isn’t happy, nobody’s happy.
• Options located on web sites and are stated to be “best practices” should not be taken as gospel.  Research and verify everything.

Successful migrations and optimization on Exadata starts with good planning

• Create top 10 out of group’s goals.  If a goal appears out of context, discuss, find out what is important about the goal and if there is another option for attaining it.
• Create a simple SOW for the optimization exercise.
• Clearly state what will be included and what WILL NOT.  Ensure all stakeholders have signed off on the SOW.

Dispel Exadata myths.  I was told by one client that, “Tom Kyte said at the last Oracle Open World that parallel should never be used on Exadata…”  Although I was sure they had heard him incorrectly, I wanted to see the original slides so that I could show them the point he was actually trying to relate.  The outcome of a quick tweet in search of Tom’s slides from OOW resulted in a bizarre and interesting set of emails/tweets when I was hit with the “140 char misconstrued tweet” issue.  Yes, I am now responsible for a “Tom Kyte says you should never use parallel with Exadata” myth…sigh…

When you are making recommendations

• Backup everything with data.
• IF you come to me with proposed parameter changes, (the silver bullet syndrome…)  please let me apologize for my lack of enthusiasm.  Even when parameter changes made a big difference in earlier versions of Oracle, it was commonly only a 10% improvement, where with SQL rewrite, I could get at least 80%-  you choose…
• Inspect opportunities for SQL rewrite- ensuring others that often it is required as systems grow and change, not because someone wrote, “bad code”.  Pointing fingers is a waste of time and unprofessional.
• Are there opportunities to incorporate parallelism.
• Is plan stability an issue? (statistics, dynamic sampling and profiles.)

Truth be known:

• Lots of SGA is not always your friend in Exadata, (if it runs in memory when it would better be served by offloading to a cell node due to high SGA, is it a good thing?)
• Do not count on storage indexes.  The feature, when implemented by Oracle is one requiring the choice of a complicated set of algorithms.  They can be your savior or your devil.
• More parallel is not better parallel.  Find you “sweet spot”.  Most environments have a setting of DOP, (degree of parallelism) that works best with over 90% of the environment’s SQL and delivers over 90% of the overall performance improvement.  The DOP is also often much LOWER than many would guess, (how often 4 has been my magic DOP… )
• Know when it’s time to implement classic warehouse/mart strategies such as star schema, bit map indexing, partitioning, mviews.  The harder you make the Exadata work now with these kinds of tasks, the less smarter it’ll be working as the environment grows.

Last note.  Although for some, it may not seem believable, but if I hear one more debate about any 11g database and optimizer_index_caching  or optimizer_index_cost_adj parameter settings,  I may go postal…:)

See ya next week at RMOUG Training Days 2013!

While at conferences, I attend a lot of dinners/social events outside of the day time festivities and although the topics on the table are often technical, sometimes they veer off to those of each person’s interest.  Discussions with me often fall into one of my areas of interest, which happens to be history, (American, European or Russian…)  Many who know me realize that if I hadn’t ended up in the technical world, I might have become a history teacher.  The common question during these conversations is “Who is your favorite American president?”  When I answer “Andrew Johnson”, I always get a quizzical look, often followed by some hoping to correct me, “You mean Andrew Jackson, right?”  I’ll stand by my answer, stating no, although I can talk about Jackson, our 7th president for hours, it was our 17th president, Andrew Johnson that I was referring to.

So often we all hear the popular, the much conversed about, the “buzz words”.  Even if we take the topic of databases and I were enter an area with five folks who work with databases and say the following words:

“SQL*Plus, inline function, Mark Hurd and RAC”

They would all have some thoughts on most of the subjects above.  They’ve read the headlines and the white papers, even if they haven’t been introduced personally to any of it, then if I proceeded to say:

“csscan, mutual recursion, Charles Rozwat and emdiag”

I might not be so lucky and would have at least a few wondering what a few of these were.  I have a tendency to find value in what may impact our day to day world, even when we don’t realize it.  How many of us know Mark Hurd and his impact to us as DBA’s, but may not realize Charles Rozwat’s impact as he heads up My Oracle Support is a understandable.  His impact is much less visible.  History is one of those things, if you may not know about it and in turn you don’t realize how much it impacts you or you fail to repeat it…

Andrew Johnson had a tough road.  When Abraham Lincoln, aware that the civil war was turning and coming into the Union’s favor, started to look towards the future and how he was going to embrace the south back to the union.  He had taken the 7th president, (speak of the devil..:)) Andrew Jackson’s “never forsake the union” stance to heart and knew that he had quite the challenge ahead of him in his second term with the Northern Republicans wanting nothing more than the South to pay for the civil war in blood….even after the war was about to be won.  These folks were angry and they wanted the confederacy to pay for a very long time…

To address this, he chose a Southern Democrat, former Tennessee Senator Andrew Johnson, who was one of the few who had stayed loyal to the Union even after his home state of Tennessee had seceded to the confederacy.  The 1864 election upon him, Lincoln counted on the next four years to take Johnson under his wing, have him help enfold the south back into the Union successfully, without the terrible plans of vengeance planned by those around him.  Many questioned Lincoln’s choice, but no one dared to challenge him on it.  Lincoln knew that without Johnson, a Southerner, along with him in the White House, the chance of the Confederacy trusting him, would be unlikely.  He foresaw Johnson as someone who might be able to ascend to the presidency after his second term and do so successfully and trusted him only as ol’ Abe would.

Johnson was known for drinking heavily, but few knew why or why he was so against slavery.  His pro-abolitionist views had sparked many a fight with his Southern neighbors.  What most didn’t realize is that Andrew Johnson was the only president to comprehend slavery from a personal perspective.  At the age of 8 yrs old, he and his older brother were  legally sold into apprenticeship to a local tailor due to the unfortunate loss of his father and financial difficulties of his mother to make ends meet even after remarrying.  For a number of years, he and his brother endured the situation until they found a chance to escape and traveled throughout the south to escape the bounty on his head.  He truly understood some of what slaves were experiencing and understood why it could not go on or allowed to simply “die out” as some of the Confederates had claimed would occur if just left alone.  Lincoln had become aware of his history and found it incredibly moving, as well as important to the cause.

There was one thing that Lincoln, nor Johnson considered though-  Lincoln’s assassination.  This unexpected turn of history rewrote all the Lincoln’s well-laid plans.  Johnson was left in a less than hospitable environment.  The Northern Republicans expected after Lincoln’s assassination that Johnson would just go back to Tennessee, but he didn’t.  He fought and won his right to be president- one, that many of the existing cabinet had accused under whispered breaths, might be behind Lincoln’s assassination, confederate plots and any other political espionage they could consider.  They were more taken aback when he planned on going forward with Lincoln’s plans to embrace the south and bring our American brothers back into the Union.  This was unacceptable for Northern Republicans and they chose at that time to put Andrew Johnson through, what might be, the most miserable four year term any president has endured.

Not only did they lock him out of his own cabinet meetings, they proceeded to leave him locked in the middle of a tug-of-war between the North and the old-South, attempting to rebuild what it once had.  The south had very few men left to lead, but many of the ones who could, had been those that had been part of the former south.  The north wanted none of this and demanded Union generals be put in place.  Johnson fought this with vetoes and battles on capital hill.  As the congressional fights intensified, with what they viewed as a “Pro-Southern” president, (while the south viewed Johnson as a trader), The Congress passed measures so that Johnson couldn’t even elect or terminate his own cabinet officials.  This lead the way to Johnson being the only other president, outside of Bill Clinton to be impeached.  He only avoided removal from office by one vote.

Johnson has been long vilified as this president that was against the North post the Civil War and against civil liberties.  The truth was, he was one man against terrible odds that could not proceed with Lincoln’s plans or any plans for that matter.  His failure left a long line of former civil war generals as future presidents, more failures in the reconstruction of the south and left it open to the carpet-bagger area, one of the most historically interesting times in Southern history, but a time that shows its impacts even today.

Johnson’s time in office is as pivotal to American history as Lincoln’s in many ways.  We just choose to remember Lincoln with monuments and book upon book in libraries and book stores.  Johnson was remembered when he finally came home, a defeated man post his presidency to find that his home town, the one that had called him a traitor on his departure for Washington as vice-president, had hung a sign, “Welcome Home Andrew Johnson, Our Hero”.

I’m always taken aback how much conferences cost, even if they are way cheaper options than many formal classes.  Education costs money, we all know that, but considering how little it costs to attend RMOUG Training Days, I always tell people, this is one of the best training opportunities anyone can take advantage of.

I need to remind folks of some of the highlights of this great conference that I am very proud to be the Training Days director of.  This year you have new speakers like Oracle Open World’s Technologist of the Year, Arup Nanda along with so many other great speakers like Frits Hoogland, Andy Colvin and John Clarke.  These folks are going to offer some great new content, as are our great return speakers like Maria Colgan, Tim Gorman, Cary Millsap and Dominic Delmolino.  It will be held once again at the wonderful Colorado Convention Center in downtown Denver, Colorado.

Support of Oracle’s ACE/ACED program is phenomenal again this year.  If you aren’t familiar with this program, its the Oracle community’s way of recognizing the database specialists who continue to give back to their Oracle Community.  For more information about this incredible program, check out OTN ACE FAQ.  The ACE and ACE Directors that will be present at RMOUG this year:

ACE Directors:  Alex Gorabchev, Debra Lilley, Arup Nanda, Bradley Brown, Sheeri Cabral, Tim Gorman, Kent Graziano, Frits Hoogland, Dan Hotka, Peter Koletzke, Cary Millsap, Daniel Morgan, Mogens Norgaard, Kerry Osborne, Gwen Shapira, Scott Spendolini, Floyd Teter

ACE: Karl Arao, Christo Kutrovsky, Michael Abbey, Jordan Braunstein, Karen Morton, Stewart Bryson, Chris Ostrowski, Andy Colvin, Kellyn Pot’Vin, Jared Still, Mark Farnham, Michael Fons, Mike Swing, Jeff Jacobs, George Trujillo, John King, Martin Widlake

Our keynote speaker this year is Mogens Norgaard, Oak Table founder, Director at Miracle of Denmark and our favorite “Great Dane”,  will be providing us with some phenomenal insight on where the database world is heading and the impact of mobile data.

Please, also check out the University Sessions and Cary Millsap’s Mastering Trace Data on the 11th.  These are additional cost from the conference, but considering the value they provide, what a great opportunity everyone who attends this conference is offered!

February is coming up quick! Hope to see you there!