AWGtek Blog

Sharepoint PowerShell Shortcuts

2012-01-29T14:48:00.000-08:00

Command to output sorted site columns:

PS C:\WINDOWS\system32\WindowsPowerShell\v1.0> [void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")
PS C:\WINDOWS\system32\WindowsPowerShell\v1.0> set-variable -option constant -name url -value "http://servername/sites/teamsite/"
PS C:\WINDOWS\system32\WindowsPowerShell\v1.0> $site = new-object Microsoft.SharePoint.SPSite($url)
PS C:\WINDOWS\system32\WindowsPowerShell\v1.0> $site.rootweb.Fields | foreach { $fieldValues = @{ "Display Name" = $_.Title; "Internal Name" = $_.InternalName; "Value" = $site.rootweb.AvailableFields.GetFieldByInternalName($_.InternalName) }; New-Object PSObject -Property $fieldValues | Select @("Display Name","Internal Name","Value") } | Sort-Object -Property "Internal Name" | Out-GridView

refs:
http://get-spscripts.com/2010/09/get-all-column-values-from-sharepoint.html

CQWP Bug

2012-01-26T10:41:00.000-08:00

[MOSS 2007] Got the following error when I selected "Show items from the following list" and browsed to a doc lib whose display name contained a forward slash (/):
Cannot save the property settings for this Web Part. The list name is not valid. The list name should refer to a list within the specified site..."

So to workaround the inability of this SP component to recognize lists with forward slashes in them, I decided to try to filter by list ID under Additional Filters. No go again. I entered the list's QUID and no data was returned. This appears to be a bug as well.

So I settled for a Data Form Web Part hooked up to an XML Web Service configured in the Data Source Library: Port set to ListsSoap12, Operation: GetListItems; listName=GUID (including curly brackets). Had to tinker with the XSLT to get the display somewhat correct, e.g. to get the doc image to show, use <img src="/_layouts/images/{ddwrt:MapToIcon('',string(@ows_DocIcon))}"/>

Adapting the schema.xml

2012-01-18T12:27:00.000-08:00

I wanted to add a new field to the base item content type -- i.e. content type ID 0x01. I also wanted to change the display name of title. This is what worked for my schema.xml.

Copy the Schema.xml from FEATURES\CustomList\CustList to my Feature folder named after the Name attribute of the ListTemplate element.

Comment out both ContentTypeRef elements.

Create a ContentType element that likes like:
<ContentType
ID="0x0100DECA7022CD234bb1BE7E9BCFF593552F"
Name="Item"
Group="List Content Types"
Description="Create a new list item.">
<FieldRefs>
<FieldRef ID="{fa564e0f-0c70-4ab9-b863-0177e6ddd247}"
Name="Title"
Required="TRUE"
DisplayName="My Title"
/>
<FieldRef ID="{AA242460-9794-4d45-A726-938D1B8B7D72}"
Name="My Added field"
Required="FALSE"
/>

</FieldRefs>

</ContentType>

NOTE the ContentType ID structure: follows the GUID method from MSDN http://msdn.microsoft.com/en-us/library/ie/aa543822.aspx, MAKE sure to remove any hyphens!!!

Add all modified fields to the Fields element, including Title, LinkFilenameNoMenu, and LinkTitle. You can get samples from a schema.xml that you copy from a list in SharePoint Manager.

Adjust View BaseViewID="1" as needed.

SharePoint Workflows on 64bit server

2012-01-11T10:32:00.000-08:00

This is a quick start on how to get a functioning workflow compiled, installed, and running to completion. It will simply log a message to the workflow history; that's it.

Assumes Visual Studio 2008 is installed on Windows Server 2003 64bit with WSPBuilder.

File, New Project, WSPBuilder Project with Workflow

Right-click project in Solution Explorer and choose Workflow, Sequential Workflow (code) - note: I tried using Sequential Workflow with definition expressed as Xaml and user code in a separate code file, but I got errors such as "RunWorkflow: System.Workflow.Activities.EventDeliveryFailedException: Event "OnWorkflowActivated" on interface type "Microsoft.SharePoint.Workflow.ISharePointService" for instance id "405ff0ae-d08d-41b8-ab69-be69ae4ccde1" cannot be delivered. ---> System.Workflow.Runtime.QueueException: Event Queue operation failed with MessageQueueErrorCode QueueNotFound for queue 'Message Properties Interface Type:Microsoft.SharePoint.Workflow.ISharePointService Method Name:OnWorkflowActivated CorrelationValues: '. "

Drag a onWorkflowActivated control onto the designer
Set its CorrelationToken property to anything, e.g. "mytoken"; then expand CorrelationToken and for OwnerActivityName, choose Workflow1 (e.g.)

Drag a logToHistoryListActivity onto the canvas under the onWorkflowActivated1 control.
Double click its MethodInvoking property, and enter the following code into the body of the method:

            var workflowProperties = new Microsoft.SharePoint.Workflow.SPWorkflowActivationProperties();

            this.logToHistoryListActivity1.HistoryOutcome = "Here is a comment " + workflowProperties.InitiationData;

Build Solution

Right click property and choose WSPBuilder, Copy to GAC

Create a folder called MyLoggingWorkflowFeature under C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\TEMPLATE\FEATURES\

Create two xml files in this folder:
Feature.xml:
<?xml version="1.0" encoding="utf-8"?>

<Feature Id="genkey id here"
Title="Workflow to create a log something to history"
Description="Logs something to history"
Version="12.0.0.0"
Scope="Site"
xmlns="http://schemas.microsoft.com/sharepoint/">
<ElementManifests>
<ElementManifest Location="workflow.xml" />
</ElementManifests>
</Feature>

workflow.xml:

<?xml version="1.0" encoding="utf-8"?>

<Elements xmlns="http://schemas.microsoft.com/sharepoint/">
<Workflow
Name="New Task Workflow"
Description="Creates a new task"
Id="yourId here from genkey tool"
CodeBesideClass="NewTaskWorkflow.Workflow3"
CodeBesideAssembly="NewTaskWorkflow, Version=1.0.0.1, Culture=neutral, PublicKeyToken=yourkeytokenhere"
></Workflow>

</Elements>

Style "Newsletter, no line" and the DVWP

2011-12-29T20:21:00.000-08:00

Tried converting (via "convert to XSLT Data View" in Sharepoint Designer 2007) a ListViewWebPart whose style was set to "NewsLetter, no line" and it would not work. The resulting XSLT produced mangled HTML tables. So trying my luck I tried converting the plain Newsletter style using the same approach. This had better results, although, there was an odd table cell on the left, which appeared due the condition ddwrt:GetVar('NumColumns')='1' being true. This is odd, because NumColumns in my list view should be four not one. Further searching the code showed that NumColumns is set as the string length of "Columns," i.e. ddwrt:SetVar('NumColumns', string(string-length(ddwrt:GetVar('Columns')))) and Columns is set to a lone period (dot).
After analyzing the CAML of the LVWP, which should be the basis for the XSLT, I found that in the CAML version, the assignment of Columns is set in a Fields element, which according to the View Schema on MSDN contains a loop over the view fields. This explains the difference: At the end of the loop, Columns is set to "....".
So what I did to patch the XSLT was simply replace the single dot with four dots in the assignment of Columns, which took care of the odd cell, and probably some other anomalies.
There is still some more fixing up to do, e.g.:
- delete the header cell for the field that displays on its own line
- setting column headers to be sortable.
- comment out the ms-alternating class assignment because it's applied unevenly to tr tags, unless you want to convert to "No line" (instructions below)
- explicitely set the class for content cells. Designer sets them incorrectly, e.g. TD Class="{$IDACAVMC}, which is a variable that is based on a List View CAML property, which no longer applies, apparently. So, icon cell would be ms-vb-icon; title cell would be set to ms-vb-title, user would be ms-vb-user; and the text cell on its own row has already been hard coded for us by the Designer ListViewWebPart to DataFormWebPart conversion process.

Now, to transform this into a "Newsletter, no line" style, do the following:
- comment out the TR tag containing the TD class="ms-nlline"
- replace the html encoded starting TR tags (all <xsl:text disable-output-escaping="yes"><tr></xsl:text>) with actual TR tags; note: must add the closing TR tag, because it was not added during conversion. To find out where to insert the closing TR for the main row, select the row in SPD, backtrack to the last <xsl:choose> that is directly above the two repeated field name comments for the field that will appear on its own line. Place the closing TR above the two field name comments.
- add the conditional ms-alternating attribute to those TR tags.
Note: There should be two xsl:text replacements (the second one being in the vicinity of "ms-nlline" below the @Remarks=) and three conditional (if statements)
ms-alternating attributes added based on a four column table with one field on its own line: two are added to the replaced TRs, and one added to the TR above the removed ms-nlline row.

Copying Discussion Items to Document Libary

2011-12-25T22:15:00.000-08:00

The Requirement is to copy discussion items to another document library.

Code ideas borrowed from: http://stackoverflow.com/questions/468469/how-do-you-upload-a-file-to-a-document-library-in-sharepoint, http://sharepoint.stackexchange.com/questions/20216/iterate-through-discussion-list, http://www.sharepoint-tips.com/2011/11/event-handler-to-archive-items-when.html

Sample code containing the general idea:

using System;
using System.Runtime.InteropServices;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Web.UI.WebControls.WebParts;
using System.Xml.Serialization;

using Microsoft.SharePoint;
using Microsoft.SharePoint.WebControls;
using Microsoft.SharePoint.WebPartPages;

namespace WebPart3
{
[Guid("26d38752-d4ce-456e-88f6-496d213627d4")]
public class WebPart3 : System.Web.UI.WebControls.WebParts.WebPart
{
string msg1;
Button saveTitle;
TextBox newTitle;
Label label;

public WebPart3()
{
SPSite site1 = SPContext.Current.Site;

msg1 = "(1.03)" ;

}

public void saveTitle_click(object sender, EventArgs e)
{
//this.Title = newTitle.Text;
string tempprint = "";
SPWeb myweb = SPContext.Current.Web;
SPListCollection lists = myweb.Lists;
SPList mylist;
SPListItem sli;
string labeltext = "";
SPList targetList;

try
{

// SPList targetList = lists["Discussion Archive"];
// SPListItem newItem = targetList.Items.Add();

mylist = lists[newTitle.Text];
targetList = lists["Discussion Archive"];
// targetList = lists["DiscussionDocs"];

// SPQuery query = new SPQuery();
//query.RowLimit = 10;
// SPListItemCollection posts = mylist.GetItems(query);

//this. .SaveProperties = true;
try
{
labeltext += " entering foreach";
foreach (SPListItem item in mylist.Folders)
{
SPListItem newItem = targetList.Items.Add();
//copy the list item to the target
foreach (SPField f in item.Fields)
{
if (!f.ReadOnlyField && newItem.Fields.ContainsField(f.InternalName))
newItem[newItem.Fields.GetFieldByInternalName(f.InternalName).Id] = item[f.Id];
}
//copy "special" read only fields that can be written to
newItem["Created By"] = item["Created By"];
newItem["Modified By"] = item["Modified By"];
newItem["Modified"] = item["Modified"];
newItem["Created"] = item["Created"];
newItem.SystemUpdate(false);
CopyAttachments(item, newItem);

labeltext += item["Body"] + ",";
sli = item;
tempprint = (string)sli["Created By"];
}
labeltext += " existing foreach";
try
{
this.Title = "yo" + mylist.ItemCount + " by: " + tempprint;

}
catch (Exception ex)
{
this.Title = "Error3: " + ex.Message;
label.Text = ex.StackTrace;

}
}
catch (Exception ex)
{

this.Title = "Error2: " + ex.Message;
label.Text = ex.StackTrace;
}
}
catch (Exception ex)
{
this.Title = "Error: " + ex.Message;
label.Text = ex.StackTrace;
}
label.Text += labeltext + "the end";
// this.SetPersonalizationDirty();
}

private void CopyAttachments(SPListItem sourceItem, SPListItem targetItem)
{
SPWeb web = SPContext.Current.Web;

SPFolder myLibrary = web.Folders["DiscussionDocs"];
Boolean replaceExistingFiles = true;

try
{
//get the folder with the attachments for the source item
SPFolder sourceItemAttachmentsFolder = sourceItem.Web.Folders["Lists"].SubFolders[sourceItem.ParentList.Title]
.SubFolders["Attachments"].SubFolders[sourceItem.ID.ToString()];
foreach (SPFile file in sourceItemAttachmentsFolder.Files)
{
byte[] binFile = file.OpenBinary();
targetItem.Attachments.AddNow(file.Name, binFile);
this.label.Text += file.Name + " attached to target ";
SPFile spfile = myLibrary.Files.Add(file.Name, binFile, replaceExistingFiles);
SPListItem newfileitem = spfile.Item;
newfileitem["Created"] = sourceItem["Created"];
newfileitem["Modified By"] = sourceItem["Modified By"];
newfileitem["Modified"] = sourceItem["Modified"];
newfileitem["Created By"] = sourceItem["Created By"];
newfileitem["Title"] = sourceItem["Title"];
//newfileitem.SystemUpdate(false);
newfileitem.Update();

myLibrary.Update();
}
}
catch (Exception ex)
{
this.Title = "Error: " + ex.Message;
}
}

protected override void CreateChildControls()
{
base.CreateChildControls();

// TODO: add custom rendering code here.
label = new Label();
label.Text = "Hello World" + msg1;
this.Controls.Add(label);

//Create text box
newTitle = new TextBox();
newTitle.Text = "myxmdiscuss";
this.Controls.Add(newTitle);

//Create Button
saveTitle = new Button();
saveTitle.Text = "Set Web Partt Title";
saveTitle.Click += new EventHandler(this.saveTitle_click);
Controls.Add(saveTitle);
}
}
}

And here's the Powershell:

$error.clear()

[System.reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")

$site = new-object Microsoft.SharePoint.SPSite("https://site")
$web = $site.OpenWeb()

$lists = $web.Lists;

$lista = $lists['Team Discussion']
$listb = $web.Folders['MyDocLib']

foreach ($item in $lista.Folders)
{
Try
{
$sourceItemAttachmentsFolder = $item.Web.Folders["Lists"].SubFolders[$item.ParentList.Title].SubFolders["Attachments"].SubFolders[$item.ID.ToString()];
foreach ($file in $sourceItemAttachmentsFolder.Files)
{
$binfile = $file.OpenBinary()
"Attached to target " + $item["Title"]
$spfile = $listb.Files.Add($file.Name, $binfile, 1)
$newfileitem = $spfile.Item
foreach ($f in $item.Fields)
{
if (!$f.ReadOnlyField -and $newfileitem.Fields.ContainsField($f.InternalName))
{
$newfileitem[$newfileitem.Fields.GetFieldByInternalName($f.InternalName).Id] = $item[$f.Id];
}

$newfileitem["Created By"] = $item["Created By"];

$newfileitem["Created"] = $item["Created"];

$newfileitem["Modified By"] = $item["Modified By"];

$newfileitem["Modified"] = $item["Modified"];

$newfileitem.Update();

}
}

}
Catch
{

"caught a system exception"
}

$error

}

Note: Minimum permissions to execute the ps1 is local server administrator + Site owner.

~SiteCollection in ParameterBinding

2011-12-09T11:34:00.000-08:00

Here's another workaround. You would like to assign the ~SiteCollection token value (or any token) to a ParameterBinding so that your XSLT can see that value and use it say in image paths, etc.

After finding a tip <a href="http://stackoverflow.com/questions/609943/dynamically-set-the-defaultvalue-of-a-parameterbinding-in-a-dataformwebpart">here</a>, the solution may be as follows.

- Put a hidden html input control with a runat=server attribute in your content place holder, e.g. <input id="siteColVal">" type=hidden name=siteColVal runat="server">
(Note, an asp:TextBox control did not work for me).
- Add a parameterbinding like: <parameterbinding location="Control(siteColVal, value)" name="SiteColRoot" defaultvalue="">
- then of course a <?xml:namespace prefix = xsl /><xsl:param name="SiteColRoot">theooroot</xsl:param> in your XSLT

Worked for me!

Sort DataFormWebPart by "Order" Field

2011-12-08T08:38:00.000-08:00

The only way I know to do this SPD 2007 is through some manual steps. BTW, the "order" field is determined by the action taken by user, which is made available via the "Allow users to order items in this view?" option for the view.

There are actually a couple of ways:

- Drag your list from the Web Part List from the Web Parts Task pane onto a web part zone.
- Then right click on web part and "Convert to XSLT Data View" which will convert the ListViewWebPart to DataFormWebPart with all the desired attributes (such as sorting) preset.

Alternately, if you created the web part as a Custom List Form from the start (i.e. with XSLT embedded), then you can edit the CAML manually.

- First open Common Data View Tasks and set it to sort by a common field such as ID.
- Then edit the SPDataSource SelectCommand, by changing "<OrderBy><FieldRef Name="ID"" to "<OrderBy><FieldRef Name="Order""

Deploy List Definition

2011-11-22T20:03:00.000-08:00

- Start SharePoint Solution Generator 2008 (comes with Visual Studio 2008 Extensions)

- Select List Definition

- click Next; if you get an error "VSeWSS Service Error: No SharePoint Site exists at the specified URL", you can ignore and click OK

- Then enter the URL, next, select the list and finish

- Open the resultant project in Visual Studio 2008 and Build - Package solution.

- Run the generated setup.bat from command line

- You may get an error if the current server does not have the Windows SharePoint Services Web application started, if so start it, and try again.

- Ignore the errors (which you might get if this is a farm with more than one web front end) concerning feature activation as we will deploy and activate next

- Open Solution management in Central Administration and Deploy the new solution (wsp file)

- another option is from the command line using the allcontenturls flag. Note the setup.bat sets the -local flag, which only applies to single server farms. However, you can use the -local flag to deploy to only the local WFE, if global deployment isn't working, e.g. because some WFEs are down.

If you intend to use the -local flag, you will need to edit setup.bat and add the following to the LDeploy section after the deploysolution operation

echo installing feature FeatureName ...

"%SPAdminTool%" -o installfeature -name "FeatureName"

- Now you can activate the lists included with the solution individually from the command line. You can verify that the solution was deployed by checking under C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\TEMPLATE\FEATURES\ for the names of your lists

- To activate the list/feature on a given site, do: "C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\TEMPLATE\FEATURES\bin\stsadm.exe" -o activatefeature -id [get the id from the setup.bat file] -url http://urlofyoursite:port

That's it! Now the list is ready to be selected during list creation.

As far as activating the list/feature on all sites, there are some powershell solutions out there, e.g. http://bable.cybermarshall.com/2009/01/17/using-powershell-to-activate-a-feature-across-all-sharepoint-2007wss-30-sites-and-subsites/

(haven't tried it yet)

ref: http://blogit.create.pt/blogs/andrevala/archive/2008/02/17/SharePoint-2007-Deployment_3A00_-Creating-Solutions.aspx

Group Email in MOSS

2011-11-20T19:08:00.000-08:00

There are a couple of gotchas in Sharepoint 2007 you have to watch for when configuring group emails.

First, if you are trying to configure alerts to an AD security group email, make sure that the security group appears under the permissions for the list, either individually or in a group. Then you can create the alert by replacing your email with that of the security group. Although, at this time, I wouldn't know where to find this alert, if I wanted to delete it.

Secondly, there a few things to watch for when trying to send an email to a Sharepoint group. First you have to email enable the sites; do it in CA under Operations, Incoming Email Settings. Then in your group's settings, create the e-mail distribution list. After submitting you will notice below "Distribution list e-mail address:" a message "Distribution list status: Creation request pending." This means you must approve the request in CA, Operations, Approve/reject Distribution Groups. When that's done, you can email the group. However, what I had to do was verify the actual email address used as shown in Active Directory Users and Computers under the OU where Sharepoint created the distribution group. In my case, the email as shown in AD was based on the domain, i.e. mygroup@mydomain.com. Whereas in Sharepoint in the group settings, was shown mygroup@server.mydomain.com. Make sure the Windows SharePoint Services Timer service is running or else the email you send to mygroup@mydomain.com (e.g. emailing it from Outlook) may never be sent.

There is still a problem with regard to adding this group email to an email action in a Sharepoint Designer built workflow. The email does not send. The NDA email contains "You do not have permission to send to this recipient." I have tried removing connection and relay restrictions in the SMTP virtual server in Exchange System Manager. I also tried configuring Sharepoint to send via a local SMTP server that relays to Exchange. But I get the same NDA error. I have had no luck getting group email to work in the three-state workflow as well, so maybe it's related to group functionality. If I select the group from the Select Users picker in Define E-mail Message SPD workflow, I do not get an NDA, but no email is sent. If I type in the group's email address, as it appears in AD, then I get the NDA. If I use the long form email address, no email is sent, nor an NDA.

MOSS - Fix "subsite1 is already in use" Error

2011-11-05T17:06:00.000-07:00

I created a Site collection, selecting the default Team Site Collaboration template. I then logged into the new site and attempted to create a subsite "subsite1" for which I selected the "News Site" Publishing template. This produced the following error: "sharepoint server publishing infrastructure feature must be activated". So then I activate this feature in the Site Collection Features page. I then go back and create subsite1. This gave me the error "The Web site address "/subsite1" is already in use. Funny that apparently the site was created despite the non-activated feature (bug -- probably fixed by now).

To fix, just go to the site collection root site, then Site Settings, Site hierarchy, click Manage next to subsite1 (e.g.), then click Delete this site.

Now I can create that site called subsite1. Moss 2007.

Multi-threading SSIM Event Search

2011-10-02T15:36:00.000-07:00

To ensure your search is multithreaded (and thus faster) as of 4.7.4.x you must break your search into separate archives. Note, that SSIM is agnostic when it comes to archives: an "archive" is simply a directory with .edx, .key, .ndx, .sar, .tdx, .vdx files. However, an official archive will have an indexed_event_fields.txt file in it.

One quick way to do this is unzip the Archive_CLI tool and run the search from the command line, inputing archives separated by a comma, e.g.:

date;java  -server  -Xmx512m  -verbose:gc -XX:+UseConcMarkSweepGC -XX:+PrintGCTimeStamps -XX:+PrintGCDetails  -jar simsar.jar -a /eventarchive/pixarchive1/,/eventarchive/pixarchive2/,/eventarchive/default,/eventarchive/ssimlogs -q "destination_ip = \"192.168.1.1\" & (destination_port = 80 | destination_port = 8080)" -c -S "," -r events.csv  -V;date

Note the extra flags are for monitoring garbage collection. Check the threads in top to verify parallelism.

Thread Dump of SSIM Service

2011-09-24T20:09:00.000-07:00

To get a global thread dump of a SSIM service, find the log_file parameter for the service in svclauncher.cfg to determine the log file that will contain the thread dump. Then issue a status to get the PID of the service in question, e.g. the rulesvc. Then while the SSIM is performing some interesting activity, type kill -SIGQUIT <PID>. Finally, open the log file to examine the thread dump and stack traces.

Quick FTP Setup

2011-09-10T12:37:00.000-07:00

This is a quick and dirty setup guide to enabling ftp uploads for your server to take advantage of FTP file transfer speeds which are much faster than SFTP/SCP. It only applies to servers in a secured lab environment, and should never be performed on production servers or public-facing servers.

- check whether vsftpd is installed:

rpm -q vsftpd

yum install vsftpd

- configure it:

chmod 777 /var/ftp/pub

vi /etc/vsftpd/vsftpd.conf

change anon_upload_enable and anon_mkdir_write_enable to YES; add line with: ftp_username=nobody (so that will prompt for user in RedHat linux)

service vsftpd start

ftp login using user anonymous with blank password

Note:

you may need to disable iptables and selinux.

refs:

http://nixcraft.com/getting-started-tutorials/725-secure-vsftpd-ftp-permissions-anonymous-user-uploads.html

http://bashcurescancer.com/installing-ftp-vsftpd-on-rhel-50-centos.html

SSIM AD Integration

2011-08-25T10:58:00.000-07:00

SSIM can integrate with multiple AD domains (do not have to be in same forest). The documentation in the Admin guide and the official KB is sparse on how to do that though currently.

Suffice it to say I was able to connect to a Windows 2003 Enterprise SP2 (not R2), as well as a Win 2003 R2 domains, in separate forests.

The trick is to make sure those domain controllers have been assigned a server certificate (not the root cert). Verify that certificate has "Server Authentication" in the Intended Purposes column of the Certificates snapin, where it should be under Personal/Certificates. When you open the certificate in the General tab it should read "You have a private key that corresponds to this certificate."

Troubleshooting steps: test connectivity using LDP.exe from the support tools installation. Also use the following command to test from the SSIM itself: openssl s_client -connect [your DC FQDN]:636

The following guide was perfect. Make sure to note the part about exporting the cert to Base-64 encoded binary X.509 (.CER). http://www.tools4ever.com/documentation/user-management-resource-administrator/ldap_ad_secure.htm?content=7030

The Symantec KB: http://www.symantec.com/business/support/index?page=content&id=TECH123285

iptables port manipulation

2011-08-24T11:44:00.000-07:00

You may find there's no /etc/sysconfig/iptables in a CentOS install with iptables disabled upon install, but you want to redirect a port, e.g. to get port 80 working in tomcat.

Do:



service iptables start

iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080

iptables-save > /etc/sysconfig/iptables

service iptables restart

To open port 80, add the following to the *filter section before COMMIT:



-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT

then service iptables restart

Numeric IP - SSIM Representation

2011-07-22T12:11:00.000-07:00

IPs in the SSIM as of 4.7.3 are represented in both positive and negative form. To convert from an IP to this form, use the following, replacing [ipvariable] with your dotted quad IP:

SELECT CASE WHEN (IPNumber - 2147483648) > 0 THEN IPNumber -(2*2147483648) ELSE IPNumber END AS IPNUMBER2 FROM ( SELECT 16777216 *
CAST(LEFT(IPNumber1, LOCATE('.', IPNumber1, 1)-1) AS BIGINT) + 65536 * CAST(SUBSTR(IPNumber1, LOCATE('.', IPNumber1, 1) + 1, LOCATE('.',
IPNumber1,LOCATE('.', IPNumber1, 1) +1) - LOCATE('.', IPNumber1, 1) - 1) AS BIGINT) + 256 * CAST(SUBSTR(IPNumber1, LOCATE('.',
IPNumber1,LOCATE('.', IPNumber1, 1) +1) + 1, LOCATE('.', IPNumber1, LOCATE('.', IPNumber1,LOCATE('.', IPNumber1, 1) +1) +1) - LOCATE('.',
IPNumber1,LOCATE('.', IPNumber1, 1) +1) - 1) AS BIGINT) + CAST(RIGHT(IPNumber1, LENGTH(IPNumber1) - LOCATE('.', IPNumber1,
LOCATE('.', IPNumber1,LOCATE('.', IPNumber1, 1) +1) +1)) AS BIGINT) AS IPNumber FROM (SELECT '[ipvariable]' AS IPNumber1 FROM
SYSIBM.SYSDUMMY1) AS LOCXTMP0) AS TEMP2

ref: http://stackoverflow.com/questions/6695428/how-to-split-an-ip-address-string-in-db2-sql-for-converting-to-ip-number

SSIM Red Queues

2011-07-14T19:40:00.000-07:00

When you see all three left red bars completely red, and the EPS counts are 0, here are some troubleshooting tips:

- Check the Sesa Agent if it's a collector server
- SSH into the collector server in question
- "su -" (change to root)
- tail the collector log which produces the most EPS or highest spikes on this collector
e.g. "tail -200 /opt/Symantec/sesa/Agent/logs/stonegate.log"
- If you see WARN entries containing "the agent queue is full" you may need to adjust the agent queue configurations: SSIM console: System > Product Configurations > SES > STATE > SSIM Agent and Manager > Agent Configurations > Standard (or the configuration applied to this server) > Logging
- adjust the following properties as needed, e.g. to increase the queue size, flush size, decrease the queue flush time (so that events are cleared more frequently).
- Save the configuration (the agent should pick up the settings automatically)

- Check the Recv-Q and Send-Q of Netstat
- on both the archiver and the correlator do "netstat -anp | grep 100"
- look for signs of downstream bottlenecks
- Examples
- if archiver is queued up sending to correlator:10010, check the correlator queues. E.g. if the correlator is queued up sending to 127.0.0.1:10080, there may be a problem with the asset service. Check assetsvc.log for errors. Try restarting the assetsvc service, and monitor the netstat -anp | grep 100 for movement in the queues.
As db2admin, cd ~/sqllib/bin
db2 connect to SESA
db2 list applications show detail
Check the SSIM-ASSET application for signs that it is currently Executing. Let it run for a while to see if it finishes.
- on correlator, check "netstat -anp | grep 555" for large number of FIN_WAIT2
- on the correlator check the icesvc.log, assetsvc.log, and rulesvc.log for recent errors (see /opt/Symantec/simserver/logs)
- Disable any rules temporarily that contain references to lookup tables to determine whether any of these rules are causing the backlog.
- Check swap space usage on Correlator. If there is any Swap used, reboot the server.
- Check SSIM Event logs for warnings or errors; query all recent logs where product = Symantec Security Information Manager or product = SSIM System
- Do a query on Event Type ID = Conclusion Updated. Verify that not more than a few events a minute are being created
- Check top on both archiver and correlator. Look for processes whose VIRT is close to the same size as RES. This process may be overworked. The memory allocation can be increased in the /opt/Symantec/simserver/svclauncher.cfg config.
- Investigate whether simserver (correlator service) is overloaded.
- ascertain the pid of the simserver in top
- do lsof | grep | awk '{print $7,$1,$2,$3,$4,$5,$6,$8,$9}' | sort -n
- look for any unusually large files in output. e.g. .que files larger than 10MB might indicate an overworked correlator.
- Do an advanced SQL query in the UI to determine whether there are any unusually large number of incidents for particular rule: SELECT count(incident_code),incident_code FROM SYMCMGMT.SYMC_IMR_INCIDENT_LIST_VIEW group by incident_code
order by count(incident_code) desc
or: SELECT desc_id, sum(event_count) as sumevtcnt FROM symc_sim_conclusion
where modified_time >= (current timestamp - 1 DAY) group by desc_id
order by sumevtcnt desc
- Disable any rules with more than 50,000 events in one day. For example, a system rule that can cause issues if not adapted is Internal Port Sweep. Simserver process memory will be exhausted typically after 24 hours, and Queues will increase steadily in /opt/Symantec/simserver/queues/ice/input. IIRC the max is three 64MB files here. If there are more than one file, the icesvc is backed up due to the amount of event tracking updates waiting to be pushed to the database. Verify that SSIM-ICE is "UOW Executing" in db2 list applications show detail.
- Also check Events > System Queries > SSIM > SSIM System > Count of Conclusions by Rule Name
- Look for FIN_WAITs associated with large queues in a frozen state in netstat for the simserver process (get pid from `status`). E.g.:
tcp 80384 0 ::ffff:127.0.0.1:10010 ::ffff:127.0.0.1:36446 ESTABLISHED 6131/java
tcp 0 145961 ::ffff:127.0.0.1:36446 ::ffff:127.0.0.1:10010 FIN_WAIT1 -
This could indicate a memory resource problem in the simserver service or another service. Check the target port, and cross reference the admin guide with the service in question. e.g. 10010 is simserver, 55562 is icesvc, etc.
If it's really in frozen state, the process may have crashed (i.e. you see 0 EPS). If the Recv-Q/Send-Q varies and EPS varies, then this could be indicative of High IO Waits. Check the IOWait in the Statistics tile of the GUI. Also check vmstat 1. Sustained IOWaits of more than 2% could indicate the drive is not fast enough.

Install syslog-ng 3.3 on CentOS 5.5

2011-06-12T20:57:00.000-07:00

Download from Balabit: syslog-ng_3.3.0bet1.tar.gz or syslog-ng_3.3.0beta2.tar.gz (see http://www.balabit.com/downloads/files?path=/syslog-ng/sources/3.3.0beta2/source) and unpack.
to get the latest, install git:
first update the repo to obtain epels:
rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm
then yum install git
or download the snapshot from http://git.balabit.hu/?p=bazsi/syslog-ng-3.3.git;a=tree
and untar over the above.

There may be prereqs on your system.

Download these rpms and install. :


rpm -Uvh kernel-headers-2.6.18-238.el5.i386.rpm
rpm -Uvh glibc-headers-2.5-58.i386.rpm
rpm -Uvh glibc-devel-2.5-58.i386.rpm
rpm -Uvh gcc-4.1.2-50.el5.i386.rpm
rpm -Uvh zlib-1.2.3-4.el5.i386.rpm
rpm -ivh zlib-devel-1.2.3-4.el5.i386.rpm
rpm -Uvh libffi-3.0.5-1.el5.kb.i386.rpm
rpm -Uvh libffi-devel-3.0.5-1.el5.kb.i386.rpm

Then install glib
wget ftp://ftp.gtk.org/pub/glib/2.29/glib-2.29.4.tar.bz2
tar jxvf glib-2.29.4.tar.bz2
cd glib-2.29.4
./configure
make
make install
done with installing glib prerequisite

Before install syslog-ng 3.3 beta, do
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH
so that eventlog is found.

also do yum install openssl-devel
or


rpm -Uvh e2fsprogs-devel-1.39-23.el5_5.1.i386.rpm
  rpm -Uvh keyutils-libs-devel-1.2-1.el5.i386.rpm
  rpm -Uvh libselinux-devel-1.33.4-5.7.el5.i386.rpm
  rpm -Uvh libsepol-devel-1.15.2-3.el5.i386.rpm
  rpm -Uvh libselinux-devel-1.33.4-5.7.el5.i386.rpm
  rpm -Uvh krb5-devel-1.6.1-55.el5.i386.rpm
  rpm -Uvh openssl-devel-0.9.8e-12.el5_5.7.i386.rpm

also:
cd eventlog-0.2.12/
./configure
make
make install
then enter syslog-ng dir and
./configure --enable-pcre --disable-mongodb --disable-ipv6
make
make install
Note: To install with PCRE enabled, must first yum install pcre-devel
After that, some more setup:
mkdir /usr/local/var
cd /root/syslog-ng/syslog-ng-3.3.0beta1/contrib/
cp rhel-packaging/syslog-ng.init /etc/init.d/syslog-ng
cp rhel-packaging/syslog-ng.conf /usr/local/etc/
cp rhel-packaging/syslog-ng.logrotate /etc/logrotate.d/syslog-ng
chmod 755 /etc/init.d/syslog-ng
vi /etc/init.d/syslog-ng
Then change the binary to "/usr/local/sbin/syslog-ng"
Then start the service:
chkconfig syslog off or chkconfig rsyslog off as applicable

chkconfig syslog-ng on

service syslog stop

service syslog-ng start

Then edit /usr/local/etc/syslog-ng.conf
add options { threaded(yes); }; to the top, and any other configs.

service syslog-ng restart

CentOS 5.5 Kernel Upgrade and iotop Install

2011-06-12T07:22:00.001-07:00

To run iotop.py you need a kernel greater than what CentOS provides. The following steps guide this upgrade, so that you can use iotop.


cd /usr/src
wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.39.1.tar.bz2
tar xjf linux-2.6.39.1.tar.bz2
ln -s linux-2.6.39.1 linux
cd linux
make mrproper
cp /boot/config-`uname -r` .config
yes "" | make oldconfig (this updates .config to all defaults)

edit .config, adding or updating the following options:
CONFIG_SYSFS_DEPRECATED=y
CONFIG_SYSFS_DEPRECATED_V2=y

make
make modules_install
make install
reboot

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/python-ctypes-1.0.2-2.el5.i386.rpm
wget http://guichaz.free.fr/iotop/files/iotop-0.4.3.tar.gz
tar zxvf iotop-0.4.3.tar.gz
cd iotop-0.4.3
./iotop.py

Disclaimer: Perform the above instructions at your own risk.

For instructions on how to make the rpm (so the lengthy compilation can be done once for future installs), see http://www.howtoforge.com/kernel_compilation_centos
Note: will need to install rpm-build and ncurses-devel

Refs:

http://www.centos.org/modules/newbb/viewtopic.php?topic_id=8488

http://forum.openvz.org/index.php?t=msg&goto=41203&#msg_41203

http://guichaz.free.fr/iotop/

Squid Proxy Quick Start Guide

2011-05-23T20:17:00.000-07:00

First install dependency: yum install gcc-c++

Download squid, e.g. the file squid-3.X.STABLEXX.tar.gz
tar xvzf squid-3.X.STABLEXX.tar.gz
change dirs
./configure --enable-icap-client
make
make install

Comment out the allow from local network:
sed -rie '/http_access allow localnet/ s/^/#/' /usr/local/squid/etc/squid.conf
Start up squid in interactive mode:
/usr/local/squid/sbin/squid -N -d1

Add the squid IP and port (3128) to the browser proxy configuration.
Try surfing to a website; you will see:

ERROR

The requested URL could not be retrieved

--------------------------------------------------------------------------------

The following error was encountered while trying to retrieve the URL: http://www.awgtek.com/

Access Denied.

Access control configuration prevents your request from being allowed at this time. Please contact your service provider if you feel this is incorrect.

Do:
Ctl-C
sed -rie '/http_access allow localnet/ s/^#//' squid.conf

/usr/local/squid/sbin/squid -N -d1

Browse to a site.

To check the access
do: tail /usr/local/squid/var/logs/access.log

DLP "Forwarding agent unavailable" error

2011-05-15T21:07:00.000-07:00

If this error is appearing in the /var/log/mail.log of your upstream MTA, when initially setting up a DLP Network Prevent for Email in Forwarding mode, try recycling the new smtp prevent server, before trying anything else.

e.g. errors like

(host #.#.#.51[#.#.#.51]
refused to talk to me: 421 4.4.1 Fatal: Forwarding agent unavailable.  Closing connection.

Recycling the DLP email prevent server got the following architecture to finally start working:

Outlook --> (port 25) Ubuntu Postfix MTA --> (port 10025) DLP SMTP Prevent --> (port 25) Ubuntu Postfix MTA --> (port 587, TLS) GMail

The upstream MTA's main.cf was configured with smtp_use_tls=no, smtp_sasl_auth_enable=no; and with relayhost=(DLP_IP):10025

The DLP server's advanced configuration had RequestProcessor.MTAResubmitPort=25 (changed from default 10026). Next Hop Configuration was set to Forward with Disable MX lookup, and downstream local MTA's IP set in Hostnames.

Setup Postfix Ubuntu Gmail SMTP Relay

2011-05-15T18:24:00.000-07:00

Refer to http://ubuntu-tutorials.com/2008/11/11/relaying-postfix-smtp-via-smtpgmailcom/

On clean Ubuntu (natty) 11.04 (with ssh and aptitude installed, and as root)

aptitude install postfix libsasl2 ca-certificate libsasl2-modules

During Postfix Configuration Package configuration wizard
On first page, choose "Internet with smarthost"
For SMTP relay host enter: [smtp.gmail.com]:587
(brackets are to avoid mx lookups)

paste in /etc/postfix/main.cf:

smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_tls_CAfile = /etc/ssl/certs/Equifax_Secure_CA.pem
smtp_use_tls = yes

while still in main.cf, add your local IP network to the mynetworks variable

create /etc/postfix/sasl_passwd with the following contents:

[smtp.gmail.com]:587 user.name@gmail.com:password

do:

chmod 400 /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd

/etc/init.d/postfix reload

Now test:

telnet postfixserverip 25
ehlo anything.com
mail from:me@whatever.com
rcpt to:someone@somewhere.com notify=success,failure
data
subject:Test from awgtek

This is a test message

.
quit

check someone@somewhere.com's email to see if message arrived
check for errors in /var/log/mail.log

After DLP DB install, start database if necessary

2011-03-10T19:27:00.000-08:00

Was getting errors when verifying the Symantec Data Loss Prevention database. When I would type the command $ORACLE_HOME/bin/lsnrctl services LISTENER I would get The listener supports no services

This can happen if for some reason the database has not started.

Try starting the database:

$ sqlplus '/ as sysdba'
Then, enter
startup at the SQL prompt.

Once the database has started, verification succeeds!

See http://www.thegeekstuff.com/2009/01/oracle-database-startup-and-shutdown-procedure/

Notes:

If you get "TNS-12541: TNS:no listener" when running the first command above, try doing: $ORACLE_HOME/bin/lsnrctl start :)
After that, you may need to restart the Vontu services. Then you should be able to login to https://vontuIP/ or https://vontuIP:8443

Here's a quick script to restart the services on the Enforce server. Add VontuMonitor.sh if single-tier:

cd /opt/Vontu/Protect/bin

  ./VontuUpdate.sh stop
  ./VontuIncidentPersister.sh stop
  ./VontuManager.sh stop
  ./VontuMonitorController.sh stop
  ./VontuNotifier.sh stop
  ./VontuNotifier.sh start
  ./VontuManager.sh start
  ./VontuIncidentPersister.sh start
  ./VontuUpdate.sh start
  ./VontuMonitorController.sh start

Three-tier DLP Installation - The Oracle Client

2011-03-06T16:49:00.000-08:00

There is little in the way of documentation for this.

So, I unwrapped Oracle_11.2.0.1.0_64bit_Client_Lin. Start reading the docs, i.e., the Client Quick Installation Guide. Followed all the instructions in there, including creating the oracle account, etc. For user oracle creation, running oracle_prepare.sh from 11g_r2_32_bit_Installation_Tools.tar.gz works.

Then in order to get past the DLP installation wizard's Oracle page, i.e. error "sqlplus is not executable at /opt/oracle/product/11.2.0/db_1/bin", I needed to do the following for the root account:

If you did not install the Oracle client to /opt/oracle/product/11.2.0/db_1/ then perform the following.
Add the following lines to the root .bash_profile:

export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=/u01/app/oracle/product/11.2.0/client_1
LD_LIBRARY_PATH=$ORACLE_HOME:/lib:/usr/lib
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
export LD_LIBRARY_PATH

PATH=$PATH:$HOME/bin:$ORACLE_HOME

Then,
. ~/.bash_profile to set the environment.
Then,


mkdir $ORACLE_HOME/bin
 cd $ORACLE_HOME/bin
 ln -s ../sqlplus
mkdir $ORACLE_HOME/lib

Copy all *.so, and so.11.1 files to $ORACLE_HOME/lib
I also created $ORACLE_HOME/jdbc/lib and to it copied ojdbc*.jar from $ORACLE_HOME.

Be sure to:

set the Base directory and Home directory to match the ORACLE_HOME and ORACLE_BASE paths above in the installation wizard of DLP.

SSH Access to PIX

2011-01-22T13:17:00.000-08:00

First generate a public key:

ca gen rsa key 1024
or 512 or 768 for modulus size (less secure)

ca save all

ssh 0.0.0.0 0.0.0.0 inside
or outside
ssh timeout 60
passwd mypassword123

Then ssh in with user: pix

ref:
see http://www.ciscopress.com/articles/article.asp?p=24664&seqNum=5
http://www.velocityreviews.com/forums/t57033-ssh-on-pix-506e-login-name.html

WSS Inaccessible after Windows Updates

2010-10-16T08:38:00.000-07:00

After a scheduled Windows Update, several WSS sites became inaccessible. The Application Event Log contained events like:

Unknown SQL Exception 33002 occured. Additional error information from SQL Server is included below.

Access to module dbo.proc_GetRunnableWorkItems is blocked because the signature is not valid.

I got similar errors when attempting to open any of the tables of the affected content databases in the embedded SQL Server (SSMS to \\.\pipe\MSSQL$MICROSOFT##SSEE\sql\query).
Relatedly, WSS creates copious logs at
C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\LOGS whenever anything goes wrong with one of the WSS services, which fills up the hard drive. This may have had something to do with the failed update.

Anyway, after much googling and trying various solutions, some of which did more harm than good, I followed one suggestion and tried running the following command:


C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\BIN>PSCO
NFIG -cmd upgrade -inplace b2b -wait -force

After some time, the operation completed and all my sites were oparational!

Missing GPOs After DC Decommission

2010-09-25T15:05:00.000-07:00

I followed the standard Microsoft method for decommissioning a domain controller. After retiring the server from lab, to my consternation, I started getting errors such as the following, for instance, when running gpupdate /force from a member workstation:

The processing of Group Policy failed. Windows attempted to read the file \\my.domain\sysvol\my.domain\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\gpt.i
ni from a domain controller and was not successful. Group Policy settings may no
t be applied until this event is resolved. This issue may be transient and could
 be caused by one or more of the following:
a) Name Resolution/Network Connectivity to the current domain controller.
b) File Replication Service Latency (a file created on another domain controller
 has not replicated to the current domain controller).
c) The Distributed File System (DFS) client has been disabled.

I found out that all the GPOs were missing from \\my.domain\sysvol\my.domain\policies. In fact even the policies folder was missing until and I created a couple of new GPOs in gpmc.msc.
The only way to fix things now is delete the dangling referenced GPOs from Group Policy Management (for which there is no corresponding ID file in the policies folder), and recreate them. I should have either backed up the DC or the GPOs individually so that I could restore them.
I followed the decommision procedure to a tee, however, I did encounter errors which I assumed were cleared up after fixing them...but apparently not.

Other symptoms of this problem: when you right click the policy in Group Policy Management and click edit, this error message appears:

"Failed to open the group policy object. You may not have appropriate rights." "Details: the system cannot find the path specified."

Besides the missing GPOs, the NetLogon share was missing as well. This was solved by creating a folder called "Scripts" in \\my.domain\sysvol\my.domain and rebooting the computer.

gSOAP Client Setup Steps

2010-09-19T13:44:00.000-07:00

My environment is Vista x64, Visual Studio 2008

Download gSoap from here: http://www.cs.fsu.edu/~engelen/soap.html. I downloaded gsoap_2.8.0.
open command prompt and change to gsoap_2.8.0\gsoap-2.8\gsoap\bin\win32
enter: wsdl2h -c -o iptocountry.h http://www.ecubicle.net/iptocountry.asmx?wsdl

enter: soapcpp2 -c -C iptocountry.h

Create a new Visual Studio 2008 Visual C++ Win32 Console Application with Empty project selected (no Precompiled header)

Copy the following files generated above by gsoap to your VS project directory (with the .vcproj).

files to copy: iptocountrySoap12.nsmap, soapC.c, soapClient.c, soapH.h, soapStub.h

Copy stdsoap2.c and stdsoap2.h to your VS project directory from gsoap_2.8.0\gsoap-2.8\gsoap\

Include the files in your project

Add a file called myclient.c to your project with the following contents:

#include "soapH.h" // obtain the generated stub  
#include "iptocountrySoap12.nsmap"
main()  
{  
   struct soap *soap = soap_new(); 
   struct _ns1__FindCountryAsString *_myipv4address;
   struct _ns1__FindCountryAsStringResponse *myresponse;
   char text[20];   
    
   _myipv4address = (struct _ns1__FindCountryAsString *)malloc(sizeof(struct _ns1__FindCountryAsString));
    myresponse = (struct _ns1__FindCountryAsStringResponse *)malloc(sizeof(struct _ns1__FindCountryAsStringResponse));

   fputs("enter an IP: ", stdout);   
   fflush(stdout);   
   if ( fgets(text, sizeof text, stdin) != NULL )   
   {      
       char *newline = strchr(text, '\n'); /* search for newline character */      
       if ( newline != NULL )      
       {         
           *newline = '\0'; /* overwrite trailing newline */      
       }      
       printf("text = \"%s\"\n", text);   
   }

   _myipv4address->V4IPAddress = text; 


   if (soap_call___ns2__FindCountryAsString(soap, NULL, NULL, _myipv4address, myresponse) == SOAP_OK)  
       printf("Country code is = %s\n", myresponse->FindCountryAsStringResult); 
   else // an error occurred  
      soap_print_fault(soap, stderr); // display the SOAP fault on the stderr stream  
}

Compile.

If you have installed a Microsoft Windows 7 SDK and are getting "fatal error LNK1104: cannot open file 'kernel32.lib'" try adding the path to the SDK lib (e.g. C:\Program Files\Microsoft SDKs\Windows\v7.1\Lib) to your project Configuration Properties \ Linker \ General \ Additional Library Directories.

Sharepoint 2010 ASP.NET Impersonation

2010-08-24T21:31:00.000-07:00

I was getting

"Login failed for user 'NT AUTHORITY\ANONYMOUS LOGON'

when attempting to access my list linked to an external content type, a remote SQL Server Database.

To resolve the issue, I opened IIS 7 \ Sites \ SharePoint - 80 \ Authentication \ ASP.NET Impersonation \ edit, then added a domain account here that could access the SQL Server.

I'm sure this might not be the best practice, security-wise, but hey, at least now it works!

Oh, to even get as far as the above login failed error, I had to do:

Central Administration \ Manage service applications \ Business Data Connectivity Service \ [my external content type] \ Set Object Permissions, then add the above domain account.

---
Update:

Trying on another machine in a different domain now and got: "access denied by business data connectivity"
when trying to add a connection to the External Content Types.
Solved by going to Central Administration \ Manage service applications \ Business Data Connectivity Service \ Set Metadata Store Permissions, and adding my domain account with all available permissions.

ORM LINQ Style

2010-08-24T12:43:00.000-07:00

I'm rediscovering LINQ, and came across this article - http://weblogs.asp.net/scottgu/archive/2007/05/19/using-linq-to-sql-part-1.aspx - by Scott Guthrie (probably not the first time I've read this article).

Ever wonder why arrows seem to point in the opposite direction in E-R diagrams? This paragraph explains it. It ties in well with O/RM, in that the mapped objects will be composed of their children. I always tended to look at it in the direction of the diamond head as more intuitive speaking of foreign keys and all.

The arrows between the four entity classes above represent associations/relationships between the different entities. These are typically modeled using primary-key/foreign-key relationships in the database. The direction of the arrows on the design-surface indicate whether the association is a one-to-one or one-to-many relationship. Strongly-typed properties will be added to the entity classes based on this. For example, the Category class above has a one-to-many relationship with the Product class. This means it will have a "Categories" property which is a collection of Product objects within that category. The Product class then has a "Category" property that points to a Category class instance that represents the Category to which the Product belongs.

I Discover DCI

2010-08-08T01:25:00.000-07:00

I just now discovered DCI; see http://www.artima.com/articles/dci_vision.html

All I can say is, it's about freakin' time! For too long we programmers have struggled with staying true to OO, but faltering because as a model OO does not provide an adequate place to store algorithms. That's why of late, I have been putting the algorithms in stored procedures. Hey maybe that's a DCI best practice! Too early to tell. Hopefully a language comes out that supports DCI.

BTW, is Trygve Reenskaug a god or what?

Controller Example

2010-07-31T23:56:00.000-07:00

Found a good example of the notion of controller of the MVC:

they handle web requests and URLs of the request map to a controller class and a closure within the class

http://grails.org/Quick+Start

Essentialy it's the thing you need (generated automatically in this case, i.e. grails) to map a URL, crafted according to the conventions of the current framework in use, to a desired output. So http://localhost:8080/my-project/book/list maps to an output containing a list of book items.

Using WDS for VMware Server Deployment

2010-07-25T21:02:00.000-07:00

The goal was to follow this excellent walk-through -- http://www.oakdome.com/lab/?page_id=78 -- to capture and deploy a Windows XP image (as a test) from and to a guest in the free VMware Server.

Everything went real well except for a few stumbling blocks.

First, I found out through much googling that it's acceptable, even preferred to run the WDS on a separate server than the DHCP server. Perfect. My DHCP server is a CISCO router anyway. For some reason I thought they had to be on the same server and a Windows one at that.

Then, I had to setup the guest to use the right nic. Again, I found out that I need to run Manage Virtual Networks as Administrator (on Vista) or else the Automatic Bridging options are greyed out.

Then I learned through a few threads on social technet that the reason the Windows Deployment Services Image Capture Wizard would freeze after checking the Upload image to WDS server checkbox and entering the server name is because the network connection is not setup. I hit shift-F10 and did ipconfig, and there was no network. The solution is to export the currently loaded boot image from the WDS to a temp directory. Then make sure you have WAIK (for Vista) installed on the WDS or another server to which you copy the wim file image. Then copy all the files from C:\Program Files\VMware\VMware Tools\Drivers\vmxnet to c:\temp\vmxnet. Then C:\Program Files\Windows AIK\Tools\PETools>imagex /mountrw c:\capture\capturebootimage.WIM 1 c:\image
Then
C:\Program Files\Windows AIK\Tools\PETools>peimg /inf=c:\temp\vmxnet\*.inf /image=c:\image\Windows
Then C:\Program Files\Windows AIK\Tools\PETools>imagex /unmount /commit c:\image
Then copy back the image, and reimport it into the boot images of WDS, naming it something like "Boot image with VMXnet drivers." Then when you PXE boot choose that image from the menu.

This same process has to be done for the deployment image. But this time, since you are using the Vista boot image, the command has to be run from a Vista or later OS apparently, or else you get PEIMG failed to complete the operation with status code 0x800703fb. when attempting to inject the drivers. I managed to inject the drivers by installing WAIK on a Windows 7 Ultimate VM, with VMware tools installed, from which I injected the vmxnet drivers. Be sure to do c:\windows\system32>imagex /mountrw c:\temp\boot.wim 2 c:\image -- Note the "2." Evidentally, there's a second image in the boot.wim that came with the Vista sources, which contains the WDS image. Note that the boot.wim file was copied from the Vista 32 DVD sources NOT the Windows 7 sources. The latter will not work.

Set clientJsLocation when using HTML_AJAX in XAMPP

2010-07-06T21:48:00.000-07:00

Please remember to set clientJsLocation, e.g.:


$server->setClientJsLocation('C:/xampp/php/PEAR/data/HTML_AJAX/js/');

Or else, you will get an error like: Unable to find javascript file: .\pear\data\HTML_AJAX\js\HTML_AJAX.js

HTML_AJAX 0.5.6 with XAMPP 1.7.3.

The only thing I can think of that causes this is reinstalling HTML_AJAX-beta via pear install? Otherwise, I guess it's understand we should always set the path. Now you know. (speaking to self)

WSS 3.0 Incoming Email -- Must use FQDN

2010-07-02T13:40:00.000-07:00

I wanted to be able to send email from my Outlook to a WSS discussion list. After much testing and troubleshooting with various configurations (and a lot of googling) I came to realize that (barring any workarounds that I'm not aware of) the CA - Operations - Incoming E-Mail Settings - E-mail server display address MUST match the machine FQDN! It cannot be the machine's DNS name. I tried adding the DNS name of the machine here, and email would just get bounced back (or get stuck in the Queue since I didn't have the smart host configured and this is not an Exchange server).

So as a result, I can send email using discussions@mymachine.fq.dn -- but NOT discussions@mymachine.dns.name.

I'm sure there are workarounds, but they might involve fiddling with Exchange servers and/or DNS servers that are outside of my control.

Now that I think about it, it makes sense that the WSS doesn't understand the DNS implications; why should it? On the other hand, why can't WSS figure out that mymachine.dns.name is the local server? Hmmmmm. Perhaps it's an IIS 6 SMTP limitation...

MyEDB - NO Indexes to be Managed

2010-06-05T11:48:00.000-07:00

This is because in My Entity Database all Entities are defined in a single index table called entity, which provides the link between the types and properties. The actual values of an entity reside in type tables which themselves are all indexed on their value column.

So, no indexes to add and manage. That eliminates probably 50% of a DBA's duties...

ref: http://myedb.blogspot.com/2010/06/myedb-no-indexes-to-be-managed.html

Errors when adding FormView to SmartPart with AJAX

2010-06-04T07:28:00.000-07:00

When I added a FormView control embedded in an UpdatePanel in a User Control, in a SmartPart with AJAX I got the following error when attempting to switch to Edit mode:

Sys.InvalidOperationException: Could not find UpdatePanel with ID 'ctl00_ctl00_xxxx'. If it is being updated dynamically then it must be inside another UpdatePanel.

I never did find a solution to this problem. But I found that if I cleared out the web parts out of the Web Part zone, then readded the smart part that appears to have fixed the issue.

I did run into other issues when adding multiple AJAX-enabled SmartParts to a Web Part zone, such as the entire zone blanking out other than the one control when switching to Edit Mode. Again, clearing out the controls and re-adding them, seems to resolve these issues, although I fear they'll crop up again...

The Original MVC

2010-05-23T11:38:00.000-07:00

The MVC is the object of somewhat of a religious ferver in the web development community. Many frameworks -- e.g. Zend Framework, CodeIgniter, etc. -- appear to me to be attempts at literal instantiations of the paradigm. Those frameworks have little appeal to me because they lock the developer into this single way of doing things. When the inevitable time comes that that way is not appropriate, the developer must resort to the framework's many workarounds, e.g. helpers or plugins (to plug the leaks? :)), extended classes, or hacks.

I wondered if the MVC is being used as it was originally intended so I researched it and read some of the creator's comments.

The MVC wasn't even called the model-view-controller; originally it was the model-view-editor. Additionally it was conceived by Trygve M. H. Reenskaug to solve the problem of network problems of the yards of the Aker Group in Norway. I think the idea got over-subscribed to by wannabe developer purists and their disciples, and as a consequence was bastardized into the impractacal forms we see depicted today around the internet.

Now, I admit, I haven't done complete dilligence on this research and so my conclusions may be premature or inaccurate. But when I see what I perceive to be serious inconsistencies between a implementations/interpretations of a model, I have to question whether the model as we know it is itself in its purest form.

For example, the Zend Framework, in which I'm no expert, apparently presupposes a single view delivered to a user at one time. If the model data must be aspected on the page, the developer has to use helpers or the ActionStack or some other hack to get the data on the page. I think this is a too-strict, too-literal, and consequently untenable, implementation of the MVC.

By contrast, the author of MVC appears to indicate multiple views presented can be the norm:
A user task may commonly be better performed with several Views present on the screen at the same time. The user will want to manipulate these Views through pointing at them, through selection in menus, or through some other means. Commands like selection typically apply to several Views at the same time. The purpose of an Editor is to establish and position a given set of Views on the screen, to coordinate them and to provide the user with a suitable command interface.

Some comments by Reenskaug lead me to wonder whether the MVC or MVE is applicable to web development at all:
Conversely, a view should never know about user input, such as mouse operations and keystrokes. It should always be possible to write a method in a controller that sends messages to views which exactly reproduce any sequence of user commands. http://heim.ifi.uio.no/~trygver/2007/MVC_Originals.pdf
How can we conclude that a PHP script can be an implementation of a "view" then? Is this what Reenskaug meant when he referred to the "perverted variants" of his model found in texts? http://heim.ifi.uio.no/~trygver/

CodeIgniter versus MyEDB

2010-05-16T20:17:00.000-07:00

CodeIgniter like most CMS's and their underlying frameworks assumes entities are mapped to tables in relational databases. MyEDB presumes the MyEDB model of data (http://www.myedb.com).

CodeIgniter like most (PHP based) CMS's assumes the view (in the MVC pattern) is composed of templates in the form of PHP files (some CMS's use non-PHP files but are nonetheless PHP interpretable in some form) which are passed parameters ala Smarty. MyEDB uses XSLT.

CodeIgniter allows the user/developer to invoke a single entry point to the system through a single controller class action pair. MyEDB allows the client to invoke a series of class action pairs, in essence multi-action controllers (maybe like Spring multiactioncontrollers? but more specifically multiple controllers) to be executed iteratively in sequence through the HTTP posting of appropriately named array variables, specifically class method pairs.

In CodeIgniter (and Zend Framework, and CakePHP, and... you get the picture) the view is an intrinsic component of the controller (defeating the notion of controller view independence). In MyEDB, the "view" is abstracted from the "controller" by a series of processing steps (see image below), ensuring total flexibility. In short, in MyEDB you are not chained down to a single way of doing things from an MVC perspective!

MyEDB Source
myedb.com

Uploading InfoPath Form Templates

2010-05-06T18:33:00.000-07:00

So, I'm working through Sharepoint 2007 for Dummies, and I'm on page 200, and I go to upload my test form, and get The restricted trust level is not supported. Quickly googled, and found http://sharepointknowledgebase.blogspot.com/2009/03/restricted-trust-level-is-not-supported.html. So I follow the fellow's excellent advice (Tools - Form Options - Security and Trust - Domain) and the error goes away.

But it can be that easy. I just get a new error (what do I expect right?): The form template is not browser-compatible. So google around again, and finally decide to do the following: Tools - Form Options - Compatibility - (check) Design a form template that can be opened in a browser or InfoPath. Click Ok. Then an error shows up in my form. It's the combo box, apparently not supported on the web. So I delete it; error goes away.

Save the form and try uploading again, first clicking Verify. Now it says This form template has not been correctly published to be browser-enabled. Open the form template in InfoPath Design mode, and click Publish Form Template in the Design Tasks task pane. Follow the steps in the Publishing Wizard to republish the form template, and then try again. So I do as instructed, choosing to "Publish this form template as a template in a document library." Published it. Still get the same error. Then, I tried selecting "To a network location" and leaving the third screen in the Publishing Wizard blank per http://www.novolocus.com/category/sharepoint/infopath/, Next, Ok, Next, publish. This saved the form again. I then tried verifying the form once more in the Upload Form Template page and it verified...I then uploaded it, got another error: The timer job for the operation has been created. However, it cannot be run because the administrative service for this server is not enabled. To run the timer job, use the stsadm.exe command line utility (stsadm.exe -o execadmsvcjobs). Googled again and got http://vspug.com/ssa/2007/10/22/a-deployment-or-retraction-is-already-under-way-for-the-solution-publishing-infopath-form/. So I start the Windows SharePoint Services Administration service, run the stsadm.exe -o execadmsvcjobs command, check the Manage Form Templates page, but see that my form is "Installing". Found this: http://www.infopathdev.com/forums/p/8034/29193.aspx, tried the suggestions, including turning on my other sharepoint server in the farm that happened to be off, but that didn't fix it. I was able to delete the "Installing" form templates through the timer definitions page and subsequently through the Manage Form Templates. Still need to figure out why it won't finish "Installing". Later, I noticed that the second Sharepoint web frontend server is low on disk space. Checking how a 25GB drive can fill up so quickly, I used windirstat and found that C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\LOGS is taking up 21 GB of space. Huh? So I delete all those log files and try uploading again. Still getting "Installing" forever....
Finally I decided that the second Sharepoint server, which is sitting on a VMware Workstation guest, needs to be removed from the farm as it appears to be experiencing issues, e.g. the ever-increasing logs mentioned earlier (containing entries like "Tracing Service lost trace events," as well as a failing system clock. As soon as I removed the second server, and attempted the template upload, the results were successful.

Replacing Form Action Scripts with sed

2010-05-01T20:24:00.000-07:00

I needed to replace all form action scripts in all my XSLT templates, specifically to make the script relative to the current path. So I did:
find . -name "*.xsl" -type f -exec sed -i "s#$[\"|'|>]$/index.php#\1index.php#g" {} \;
There are a few of things to notice about this command:

I used a double quote and not a single quote for the sed script namely because A single quote may not occur between single quotes, even when preceded by a backslash. *

The parenthesis required backslashes because sed apparently is not running in extended mode on my machine.

The argument to sed needs to be quoted to avoid being parsed by the bash shell interpreter. Aside the from the parenthesis the only other character needing escaping is the double quote in the regex. Even the \ don't need further escaping because a double quote string preserves the literal value of all characters except \, $, and `. The backslash has its special meaning only if followed by ", $, `, \, and newline. *http://answers.google.com/answers/threadview/id/496910.html

Use the User DSN When Debugging

2010-04-09T14:53:00.000-07:00

Couldn't figure out why I kept getting "failed to call the odbc driver connection utility" when attempting to configure data tables in the DataSet designer in Visual Studio 2005 on a Windows 7 machine -- a duo that has incompatibility problems apparently.

I finally realized that the problem was that I had created a System DSN (like I always do) for the ODBC connection. I created a corresponding User DSN and it worked like a charm.

Virtual Host in Tomcat and NAT

2010-04-02T15:55:00.000-07:00

Well, I didn't figure out how to separate the apps by port, but I did learn that it's possible to do so by IP -- by setting the name attribute in the Host tag of server.xml.

The problem appeared however when I attempted to port forward port 80 to the second host entry in the m0n0wall NAT configuration. Browsing to the site from outside the firewall kept bringing me back to the first host entry which by default was set to localhost. That will never do, because I don't want external users seeing the Tomcat management page or welcome page for that matter. That was the whole point of creating a separate app in a separate location on the hard drive (which was done by the way via setting the docBase attribute of the Context tag which goes under the Host tag. I was creating an Open Blue Dragon or openbd application and I didn't want it sitting under the default webapps virtual host.

I solved the problem by simply setting my new app, the second host in server.xml, to localhost, and assigning the default app a second IP (from a second NIC) for management purposes. I'm not sure why or how, but this fixed the problem. I can now access my ColdFusion web app from outside the firewall through port forwarding. And the managment app is only visible from the server itself.

Don't forget the Share Permissions...

2010-04-01T08:04:00.000-07:00

I was getting errors when attempting to access a Visual Source Safe DB configured as a share from the network:


error: Microsoft Visual SourceSafe
Unable to open user login file ....user.log

I thought how could this be, my computer user account is in the Administrator's group. I even tried copying a file to the ...data\loggedin folder, and got this error:


Destination Folder Access Denied
You need permission to perform this action

Silly me. The sharing permissions were not set up correctly.

To fix, open sharing on the VSS DB folder. Click Permissions. Add the Users group, and assign Full Control.

MIssing BIN DLLs from VS Website References

2010-03-30T13:55:00.000-07:00

If you are ever in the predicament where after copying over a Visual Studio 2005 website to another machine and are not finished working out the dependencies, here's a tip: Make sure all the assemblies listed in the "add assembly" tags in the web.config can be found on the current machine first, or else none of the Bin folder DLLs will be added as references, no matter how hard you try.

I ran into this problem, and Visual Studio 2005 refused to add the Bin folder references, until those web.config assemblies had their match in the GAC. Go figure!

MyEDB SQL One Liner

2010-03-16T21:20:00.000-07:00

In this case the use case is to reduce the quantity of each product variation ("ron" in myedb speak), i.e. items in a shopping cart, when the total transaction is finalized. The following code can be placed in a transaction inside a MySQL 5.x stored procedure:


update property natural join int_prop natural join entity_property natural join entity 
inner join medb_transaction on entity.eid = medb_transaction.eiddst and entity_property.ron = medb_transaction.ron
and property.prop_group_id = medb_transaction.prop_group_id
set int_val = int_val - qty
 where medb_transaction.eidsrc = 8  and property.prop_id = 15

The prop_id 15 is the ID of the 'quantity' property. eidsrc is passed as a parameter to the stored procedure and represents the user id (or "eid" for entity id in myedb speak ;) ).

New myedb.com Domain for Framework

2010-03-10T20:01:00.000-08:00

My first post to describe my MyEDB (My Entity Database) framework/CMS was placed on a holding site at netfirms:

This is my stand-in site to promote my concept of an "Entity Database" which is more than a relational database -- it gives the DB designer more flexibility in setting up and tearing down entities. I call it and the supporting code framework MyEDB for My Entity DB.

It does away with the traditional notion of entity relationships, and just focuses on pure entities, no relationships and the headaches they cause Database designers. The purpose for relationships has evolved from an ingenuous desire to model entities and their relationships into being merely an optimization device. My framework gets rid of the pretense. All optimization is stored natively through the use of single type tables.

One added benefit of this static architecture is that of being essentially a low level database per se, but one that is accessible to database designers.

Combine this with an elegant PHP-based RESTful MVC framework, and all that's left is defining inputs and outputs and coding them. And the effort of coding the input/output wire-ups simply scales to the level of your requirements -- i.e. pay per requirement.
Requirements can be defined as any of the following:
- Create a form for Inserting, updating, or deleting an entity (e.g. a customer record)
- Create a new view of the entity, purely for display (e.g. a product, or any entity. Examine one of your Excel spreadsheets rows for examples of "entities")
- Build search functionality for all your entities or only a subset

Connecting to Web Service with .NET

2010-02-24T06:19:00.000-08:00

I needed to connect to a remote Java WSRF Web Service having access only to the WSDL and login credentials. At first I tried using Visual Studio 2008, but found it doesn't support old style WSE so well, so had to revert to Visual Studio 2005, over which I installed Microsoft WSE 3.0.msi.

I then pointed wsdl.exe at the https://url?wsdl to generate the proxy class, which I put in the App_Code folder of a test web project. wsdl.exe caused the Service class to inherit from System.Web.Services.Protocols.SoapHttpClientProtocol. I had to replace this with Microsoft.Web.Services3.WebServicesClientProtocol to get the proper authentication interface.

I then had to manually add a wse3policyCache.conf file to get the policy setting intialized (see https://www-secure.symantec.com/connect/forums/newbie-help-adding-security-header-request-object#comment-3625021).

Then, following guidance in http://bytes.com/topic/net/answers/560786-how-add-soap-header-soap-message, I added a SoapHeader class as well as an attribute to the target proxy class method.

The SoapDocumentMethodAttribute Action attribute was set to a blank string so I had to set that to the fully qualified url of the method from the WSDL, i.e. .../PortType/TheMethod.

In the client code, I then just had to call myproxy.SetClientCredential(..), then I could use the proxy class without problems.

Resize Root Filesystem (/) with LVM

2009-12-06T16:51:00.000-08:00

I made the mistake of accepting the default LVM options during a Debian Lenny installation, only to find out that the root / had filled up quite quickly, having started out with only around 300mb.

I wanted to transfer some free space from /home to /; so did as follows:

Select filesystem (e.g. /dev/mapper/debian-home ... 2.5G) to shrink from Filesystem column in df -h
I chose /home, so must be ssh'd as root; use 'who' command to verify no other users are logged in and have the home dir open.

Resize /home:
#umount /dev/mapper/debian-home
#e2fsck -f /dev/mapper/debian-home
#resize2fs /dev/mapper/debian-home 1G
select the corresponding volume to shrink from lvdisplay, i.e., the LV Name field (e.g. /dev/debian/home)
#lvreduce -L -1.3G /dev/debian/home

Extend root volume (/):
verify there is Free PE in vgdisplay. Check this value against the following lvextend command size option
To extend root volume (/), find it's coresponding name by looking up [LV Name] (e.g.
/dev/debian/root) containing 'root' in lvdisplay
then:
#lvextend -L +1G /dev/debian/root
resize the root file system. Find out the Filesystem name for the / mount by checking df
# resize2fs /dev/mapper/debian-root
check df to verify / is resized.
remount home:
#mount /dev/mapper/debian-home

Warning! This procedure is very risky. A full backup of the system immediately before resizing volumes is strongly advised on production systems. After the volumes are resized, run fsck to fix any errors.

refs:
ref for shrinking: http://www.linuxquestions.org/questions/linux-enterprise-47/shrink-lvm-without-dataloss-557746/
ref for extending /: http://www.linuxquestions.org/questions/linux-general-1/lvm-resizing-the-root-partition-361663/

Vertical List Oriented Columns with XSLT

2009-11-27T15:00:00.000-08:00

The general idea is snake your list of items a set number of times down each column. So for a list of 7 items (XML siblings) and "n-rows" being set to 3, the resulting columns will look like this:

item 1	item 4	item 7
item 2	item 5
item 3	item 6

The XSLT code:

<xsl:variable name="n-rows" select="3"/>

<xsl:template name="recurse-rows">

<xsl:param name="index" select="1"/>

<xsl:param name="modrow" select="1" />

<tr>

<xsl:for-each select="/root/item[position() mod $n-rows = $modrow]">

<td>

<xsl:value-of select="value" />

</td>

</xsl:for-each>

</tr>

<xsl:if test="$index &lt; $n-rows">

<xsl:call-template name="recurse-rows">

<xsl:with-param name="index" select="$index + 1"/>

<xsl:with-param name="modrow" select="($index + 1) mod $n-rows"/>

</xsl:call-template>

</xsl:if>

</xsl:template>

sources for ideas:
http://www.stylusstudio.com/xsllist/200407/post30810.html
http://www.perlmonks.org/?node_id=518095

Tips for when Setting up WordPress

2009-11-15T13:23:00.000-08:00

Platform: Ubuntu 9.04/Apache 2

To set up WordPress as a virtual directory under a VirtualHost, add an alias to the host file in /etc/apache2/sites-available:

e.g.:

<VirtualHost 172.16.134.5:80>

  Alias /blog/ "/usr/share/wordpress/"

  ServerAdmin webmaster@localhost
 DocumentRoot /home/wordpress/public_html
 DirectoryIndex index.php

  <Directory "/usr/share/wordpress/">

  Options Indexes FollowSymLinks

  AllowOverride All

  Order allow,deny

  Allow from all

  </Directory>

  ...

</VirtualHost>

The guide I used was https://help.ubuntu.com/community/WordPress.

Note: where you run the command "sudo bash /usr/share/doc/wordpress/examples/setup-mysql -n wordpress localhost" you MUST change localhost to the IP or DNS name if you are setting up the site on a remote server. The reason, as I found out the hard way (I use "localhost" at first) is that the script creates a file under /etc/wordpress called config-<hostname>.php. Then when you go to run the web installation routine, the application can't find the config file, unless you set <hostname> in the above command to match exactly the domain used in the URL. This is because the config file is constructed in /etc/wordpress/wp-config.php using $_SERVER['HTTP_HOST'].

Dell Memory Errors with BIOS USB Support

2009-11-01T09:35:00.000-08:00

My 2550 Dell Server was displaying memory errors. It would split the amount of memory available and errors would be produced in the Windows Memory Diagnostic utility starting with the MATS+ test. It also mentioned something about an invalid NVRAM configuration in the startup messages. Dell Diagnostic utility was not helpful. I started disabling all options in the Integrated devices section of the BIOS. I found that by disabling BIOS USB support, the memory errors went away. Go figure!

Change Directory with String Replacement

2009-10-17T12:42:00.000-07:00

Found out through experimentation how to change directories in the bash shell using sed string replacement. I need a quick way to switch from production to staging sites which have the same path but for the user home directory.

First I tried:


echo `pwd` | echo `sed -e 's/production/staging/' ` | cd

That didn't work.

Then I tried:


cd $(echo `pwd` | echo `sed -e 's/production/staging/' ` )

That worked. Go figure...

Add Hyperlink using jQuery

2009-10-04T11:53:00.000-07:00

You can test this out in Firebug. Say you have a td element, <td id="foo" />,
you can add a link to this using jQuery:
jQuery("<a />").attr("href", "http://yahoo.com").appendTo("#foo").text("link to yahoo").after("<br />");

WSS 2.0 to WSS 3.0 Migration

2009-09-28T10:04:00.000-07:00

I had a WSS 2.0 site (no custom parts) that I needed to upgrade AND migrate to another server. So I pretty much followed the advice on http://geekswithblogs.net/redwards/archive/2007/10/26/116365.aspx .

The above article is unclear about how the DB was transferred however, so had to wing it. Basically what I ended up doing was copying the .mdf and .ldf files of the WSS 2.0 content DB over to the Data folder of a SQL Express 2005 server --NOT the WSS 3.0 embedded DB (I tried that, but it gave me errors when I ran stsadm).

Note, after the DB was added and all set up in the Central Administration, I had to change the site collection administrator. The migrated site still had the OldServer\Administrator as the Primary site collection administrator. Other than that pretty seamless...

Getting memcached to Run at Startup on Ubuntu

2009-09-20T01:00:00.000-07:00

For some reason memcached would start and then die shortly after on reboot. This is on an Ubuntu 8.04 with updated packages for Apache2, PHP5, and memcached installed through apt-get.

The only way I could fix the problem was to remove memcached from the rc scripts:


update-rc.d -f memcached remove

Then add the following line to /etc/rc.local:


/usr/bin/memcached -d -p 11211 -u nobody -l 127.0.0.1 -m 64

I'm sure this will have to be adjusted if I ever need more memcached processes or if I need the PID file. Speaking of which, I suspect that the canned scripts that came with this memcached 3.0.1 from Ubuntu are too convoluted with regard to the PIDs, which already have problems of their own.

This bug report set me in the right direction.

Eclipse Simpletest plug-in Path errors

2009-08-24T21:09:00.000-07:00

I struggled with this one until through experimentation I found out the problem. I have the Simpletest plug-in 0.2.5 for Eclipse.

I had


require_once dirname(__FILE__) . '\MyClass.php'; 
class test extends UnitTestCase {
 function test_pass(){
  $tc = new MyClass();
        $boolean = $tc->my_method() == "hello";
        $this->assertFalse($boolean);
 }
}

But when running, I kept getting:


PHP Fatal error:  Class 'MyClass' not found in ...test.php...[line number]

I finally found out through trial and error that to fix this fatal error I simply needed to replace the opening short tag with the full php tag <?php . That means I'm going to need to do this to all my class files. Might as well make that a best practice from now on.

IIS 7 and Permissions Quirks

2009-08-23T20:45:00.000-07:00

I just enabled IIS 7 on Windows Vista SP 2 and enable FastCGI. I create a new website and point it to a folder under my Documents folder with a test.php file. I open the page http://localhost/test.php in IE and get:

HTTP Error 401.3 - Unauthorized
You do not have permission to view this directory or page because of the access control list (ACL) configuration or encryption settings for this resource on the Web server.

Ok, so I add IIS_IUSRS to folder ACL, but that doesn't fix it. I then added the Users group to the ACL and that fixes it. Strange indeed because my user account is already in the ACL and I'm in the Administrators group, which is also already in the site folder ACL. Now I just have to wonder whether this is a Windows bug, or whether there's some other user under the Users group that's being impersonated by some process. Hmmmmm.

Use encodeURIComponent in AJAX

2009-08-09T11:59:00.000-07:00

In the XMLHttpRequest "send" method it's best to encode the parameters using the Javascript function encodeURIComponent as opposed to escape or encodeURI. For details why, see
http://xkr.us/articles/javascript/encode-compare/ .

I learned this by researching why the solution at http://www.captain.at/howto-ajax-form-post-request.php did not work for me. For example, if I use escape, the '+' symbol converts to a space in the database; and if I use encodeURI, any '&' will cause all subsequent characters to not make it into the database -- because those characters aren't converted with the other two Javascript methods.

Hacking CAPTCHA

2009-08-08T12:23:00.000-07:00

One way to hack one type of CAPTCHA is as follows:

Assuming the web app sets the captcha code in a session variable and assuming it only checks it for equality, you can use Paros Proxy to trap and alter the captcha url in the image tag (assuming, once again, this is the method used to display the image).

First clear all of the target website's cookies (including session cookies -- still not sure how to do this in IE), then open the target web page form with Paros running with "Trap response" checked in the Trap tab. Then edit or remove the link in the image tag to the captcha image generator, then click Continue. When the page is resolved the captcha image will appear broken, indicating the script that produced it never was run for this session and hence no security session variables were set. Finally, leave the captcha input box empty and submit. If all goes well the form will submit successfully.

The way to protect against this kind of attack is to set another secret session variable in the captcha script then check for it in the form validation.

(this advice is provided to assist newbie web developers with no guarantees. Malicious hackers and script kidies are not welcome.)

udp_scan on Debian 4

2009-07-04T20:11:00.000-07:00

Download & Setup:


lftp -c 'open -e "mget port-scan*" ftp.porcupine.org/pub/security'
tar zxvf port-scan.tar.gz
cd port-scan
make CFLAGS=-D_BSD_SOURCE

There will be some errors.

Then, run udp_scan on target:
./udp_scan 10.10.10.10 1-1024

Note: udp_scan also comes with the SATAN package (and its successors).

Installing SATAN 1.1.1 on Debian 4.0

2009-07-04T13:29:00.000-07:00

wget http://www.ibiblio.org/pub/packages/security/Satan-for-Linux/satan-1.1.1.linux.fixed2.tgz

unpack in /tmp and cd to satan-1.1.1

edit reconfig:


sed -i 's/version 5/v\(ersion \)\?5/' reconfig

if no dev environment:


apt-get install make 
apt-get install build-essential

then:


perl reconfig
make linux

run:


./satan

vtund Experimentation part 3 - Bridging

2009-07-01T11:26:00.001-07:00

Picking up from "vtund Experimentation part 2" we are going to set up the bridge interface, which will bridge tap0 and eth1 as depicted in http://blog.kovyrin.net/2006/04/05/connecting-two-remote-local-networks-with-transparent-bridging/ (although there it's eth0).

On both server and client sever the vtund connection (if any) and kill the vtund process.

On the server:
in the lion config of vtund-server.conf, replace
"ifconfig "%% 10.1.0.1 netmask 255.255.255.0";
with the following lines:
ifconfig "%% up";
program "brctl addif br0 %%";
comment out the route command.
change the password (recommended)

On the client:
in the lion config of vtund-client.conf, replace
"ifconfig "%% 10.1.0.2 netmask 255.255.255.0";
with the following lines:
ifconfig "%% up";
program "brctl addif br0 %%";
change the password to same as server's (if changed on server).

On the server:
in /etc/network/interfaces
remove the eth1 ip configuration, except for the line
auto eth1 # to bring up eth1
Add the br0 ip configuration:
iface br0 inet static
address 192.168.100.99
netmask 255.255.255.0
bridge_ports eth1

On the client:
in /etc/network/interfaces
remove the eth1 ip configuration, except for the line
auto eth1 # to bring up eth1
Add the br0 ip configuration:
iface br0 inet static
address 192.168.100.100
netmask 255.255.255.0
bridge_ports eth1

On both server and client bring up br0:
ifup br0

On the server:
vtund -n -s -f /etc/vtund-server.conf

Then on client:
vtund -n -f /etc/vtund-client.conf lion server_public_ip_address

With luck, a UDP connection (tunnel) will be initialized. From client you should be able to ping 192.168.100.99 (server's br0 ip address).

On the server, type
brctl show
results:


bridge name bridge id STP enabled interfaces
br0         xxxxxxxx  no          eth1
                                  tap0

Now set up another machine on the 192.168.100.x network on either side and ping both machines from it. If the machines can't ping eachother, ensure that the switch (vSwitch if in VMware) is set to Accept in Promiscuous Mode Policy Exceptions.

(see also Building Linux Virtual Private Networks (VPNs) on Google Books for example of the Ethernet Tunnel)

vtund Experimentation part 2

2009-07-01T06:40:00.001-07:00

On both client and server:
fresh stable (ubuntu LTS 8.04) Ubuntu, get all updates.
then apt-get install bridge-utils (per http://blog.kovyrin.net/2006/04/05/connecting-two-remote-local-networks-with-transparent-bridging/)
then:
apt-get install uml-utilities
tunctl -u root -t tap0
ifconfig tap0 0.0.0.0 promisc up

then apt-get install vtun

modprobe tun

Then on server, copy /usr/share/doc/vtun/examples/vtund-server.conf to /etc
comment out: bindaddr { iface lo; };
in lion configuration:
change: "compress lzo:9;" to "compress no;"
change: "encrypt yes;" to "encrypt no;"
comment out firewall lines

Then on client, copy /usr/share/doc/vtun/examples/vtund-client.conf to /etc
edit /etc/vtund-client.conf: under the "lion" configuration, change "device tap1" to "device tap0"

Then on server:
edit /etc/network/interfaces:
auto lo
iface lo inet loopback

iface eth0 inet static
address public_ip_address
netmask public_mask
gateway public_gateway
auto eth0

Then on client:
edit /etc/network/interfaces:
auto lo
iface lo inet loopback

iface eth0 inet static
address public_ip_address
netmask public_mask
gateway public_gateway
auto eth0

Then on server:
vtund -n -s -f /etc/vtund-server.conf

Then on client:
vtund -n -f /etc/vtund-client.conf lion server_public_ip_address

With luck, a UDP connection (tunnel) will be initialized. From client you should be able to ping 10.1.0.1 (server's tun0 ip address).

Read latest log entries created on Linux

2009-06-30T13:09:00.001-07:00

find / -name \*.log\* -mmin -3 -print |  xargs tail

This lists all the log entries on the machine in the last three minutes.

Ubuntu Default Gateway

2009-06-18T09:51:00.001-07:00

auto eth0 -- sets eth0 to come up on /etc/init.d/networking restart

tip: put auto eth1 before auto eth0 in /etc/network/interfaces in order to make eth0 the default gateway

vtun Experimentation

2009-06-17T21:00:00.001-07:00

So I get fresh stable (ubuntu LTS 8.04) Ubuntu, get all updates.
then apt-get install bridge-utils (per http://blog.kovyrin.net/2006/04/05/connecting-two-remote-local-networks-with-transparent-bridging/)
then:
apt-get install uml-utilities
tunctl -u root -t tap0
ifconfig tap0 0.0.0.0 promisc up

then apt-get install vtun

on both client and server:
modprobe tun

then put at end of /etc/network/interfaces (note: create bridge dynamically through vtun program commands instead?):
auto br0
iface br0 inet static
address 192.168.1.200 (199 for server)
netmask 255.255.255.0
bridge_ports eth0

then ifup br0

on client /etc/vtund.conf

options {
port 5000;
timeout 60;

# Path to various programs
ifconfig /sbin/ifconfig;
route /sbin/route;
firewall /sbin/iptables;
}

IP-Tunnel {
passwd abcd1234; # Password
type ether; # Ethernet tunnel
device tun0;
up {
# Connection is Up
ifconfig "%% up";
program "brctl addif br0 %%";
};
down {
# Connection is Down
ifconfig "%% down";
};
}


On server /etc/vtund.conf:
options {
    # type stand;
    # stand(default), inet (used only at server)
    port 5000;
    # Server will listen on this port for incoming requests.
    syslog daemon;
    # Syslog facility

    # Path to various programs
    ppp /usr/sbin/pppd; #Path to the pppd. Use "which pppd" to find this.
    ifconfig /sbin/ifconfig;
    route /sbin/route;
    firewall /sbin/iptables;
    ip /sbin/ip;
}
# Default session options
default {
    type ether;
    #tun, ether, tty(default), pipe (Used only at Server)
    proto udp;
    #udp, tcp(default) (Used only at server)
    # device tun0;
    compress no;
    # no, yes, zlib:(1-9), lzo:(1-9); e.g. zlib:1 (default) (Used only at server)
    encrypt no;
    #yes, no (used only at server)
    stat yes;
    #yes, no: check /var/log/vtund/SessionName_X
    speed 0;
    #By default maximum speed, NO shaping (Used only at server)
    keepalive yes;
    #Used to keep alive the connection. (Used only at server)
}
# TUN example. Session 'cobra'.
    IP-Tunnel {
        passwd abcd1234; # Password
        type ether; # IP tunnel
        proto udp; # UDP protocol
        device tap0;
        compress no; # lzo:9; # LZO compression level 9
        encrypt no; # Encryption
        keepalive yes; # Keep connection alive
        stat yes; #yes, no

        up {
            # Connection is Up
            # 10.3.0.1 - local, 10.3.0.2 - remote
            # ifconfig "%% 192.168.254.201 pointopoint 192.168.254.200 mtu 1450";
            # route "add -net 192.168.0.0 netmask 255.255.255.0 gw 192.168.254.200";
            ifconfig "%% up";
            program "brctl addif br0 %%";
        };
        down {
        # Connection is down
        # 10.3.0.1 - local, 10.3.0.2 - remote
        ifconfig "%% down";
        # route "del -net 192.168.0.0 netmask 255.255.255.0 gw 192.168.254.200";
        };

}

then on server, vtund -s (change port to 5000?)

then on client: vtund IP-Tunnel 172.16.2.16 (real of IP of server).

other references:
recompile kernel http://www.howtogeek.com/howto/ubuntu/how-to-customize-your-ubuntu-kernel/

check Universal Tun/tap device driver support under Device Drivers > Network device support

try tunctl to create tap, e.g. http://www.mail-archive.com/vtun-users@lists.sourceforge.net/msg00001.html

try steps here: http://adamolson.com/articles/UnixReview.com%20%20Secure%20Tunneling%20between%20Intranets%20with%20VTun%202000.htm
or here: http://www.opennet.ru/soft/vpn_table/vtund1.html

read http://vtun.sourceforge.net/setup.html

tips for manual setup: http://www.blindhog.net/linux-bridging-for-gns3-lan-communications/

and http://www.linux.com/archive/articles/54894 vtun tutorial

bridging in Linux: http://www.linuxfoundation.org/en/Net:Bridge

http://dkprojects.wordpress.com/transparent-ethernet-bridge/ - transparent bridge (hub)

TortoiseSVN - Quick Start

2009-03-22T13:01:00.001-07:00

TortoiseSVN repository set up:

-Install TortoiseSVN
-Perform these steps in Windows Explorer:
-right click in new empty dir (named after your project) and
choose ToroiseSVN > create repository here. This will be your repository, so make note of its location.
-Then rightclick your project folder and click TortoisSVN > Import, Browse to the path to your newly created repository and hit ok. Alternatively, enter a path like file:///SVNRepos/MyProject
-Rename your project folder by appending ".bak" or anything
-Create a new empty folder next to your old folder having the same original name of the project folder
-Open the new empty project folder and right click in the empty folder and choose SVN Checkout...
-Make sure the path to your newly crated repository is under "URL of repository:" and hit OK

Now you have a version controlled project!

Thoughts on Ruby on Rails

2009-03-07T21:11:00.001-08:00

Was doing some research on the pros and cons of ROR and came across this article. A snippet:

Rails is designed from the ground up to create dynamic Web sites that use a relational database backend. It adds key words to the Ruby programming language that make Web applications easier to configure. In addition, it's designed to automatically generate a complete, if somewhat crude, Web application from an existing database schema. The latter is both Ruby's greatest strength and its Achilles' heel. Rails makes assumptions about database schema naming conventions that, if followed, make generating a basic Web site a matter of executing single command. But to do this may require additional configurations or in some cases may not be possible at all. You're also likely to find that just about every database convention that Rails expects can be overridden, but the more overriding that is needed, the less productive the platform becomes.

I knew this instinctively long ago when I did research debating whether to use an existing content management system (CMS) or create one from scratch. At that time, I was visiting blogs where followers of the ROR cult would rave about ROR and attack anyone that pointed out its flaws. There was this one guy in particularly that kept pointing to the database, and noting that the database design is too restrictive. He argued in favor of his own CMS, RADICORE, from which I got the idea to incorporate XSLT into my new CMS.

My opinion is that ROR is too tightly coupled to the database. The database design dictates the application. In that sense, it's a two-tiered application. I know next to nothing about ROR, but I've got friends and associates who work with it for a living. But I can't for the life of me understand the attraction of a platform that doesn't have a robust middle layer! Is there any concept of business objects in ROR?

Another article was linked to from the above article which lays out the restrictions and implications of a rails application:

Conventions that relate to legacy schema integration with Oracle and Rails include the following:
Tables are named in using the plural form of the model they represent. For example, an "employee" model maps to an "employees" table.
All tables that contain data that will be updated contain a primary key called "id."
In the case of Oracle Database, this primary key is incremented using a sequence with a name based on the table it increments. A table named "employees" that contains an "id" will be incremented by a sequence named "employee_seq."

Apart from the unappealingness of the restictions of how one must name the tables, I find something else more disquieting. The unspoken assumption in the Rails architecture is that tables map to entities in the traditional sense, i.e. that they reflect entities that appear perhaps in the real world and directly against which the application will perform its actions.

This makes integration of rails with one of my CMS's impossible, since the tables in my "MyEntityDB" framework are really meta-tables.

Then there's all those files in the "layouts" folder. That's another unspoken assumption that the design will reside in files. That makes another one of my CMS's impossible to work with Rails, because the design parts happen to reside in database table fields.

It also appears to be a cryptic version of Smarty.
Here's some sample ROR code from a downloadable app from here:

The embedded variables indicate a strong reliance on the application environment, which also I find disquieting. I prefer pure XSLT for the view layer, since it depends on XML for its data. That's what I use in my MyEntityDB framework, because it enforced complete separation between code and presentation (not just token separation as in Smarty Templates).

Volts, Amps, and Watts, Oh my!

2009-02-21T23:34:00.001-08:00

http://en.wikipedia.org/wiki/Volt

Had to read it twice before I realized that a Volt represents the potential power, while Amps represents the actual power. "Watts" represent the actual power as well, but it's the total power arrived at by multiplying amps by volts.

Another realization I had when researching power consumption constraints for a possible home data center was that the "watts" specification on the side of appliances is actually the measure of watts per hour (DUH!). So if the space heater is 1500 watts, that means it's using 1500 watts (1.5 kW) per hour (if it's turned on and set to High).

More notes:
Helpful discussion here.